^{(Last updated: 11/11/2019 | PDF | PERSONAL OPINION | COMPANY CONFIDENTIAL)}

I went to see a movie last night by myself (R rated and wife not interested), and one scene in the “Star Wars: The Rise of Skywalker” trailer made my thought drifted back to analytics: with each vendor self-contained as a space battleship, what common internal structure they may share and how can they co-exist in the most efficient way?

The Inverted Pyramid

By effort sizes, complexities and their distance from real applications, activities in each analytics shop can be organized into an inverted pyramid structure. In the Lab, the bleeding edge stuff are tested out. In the Factory, the well-defined, shared and low-level parts are manufactured at a steady pace. In the Matrix spanned by the technologies dimension and the domains dimension, technological know-how and domain knowledge are developed, accumulated and curated, ready to be applied to apps , solutions or services offerings one level up in the Experience layer. Please note I didn’t mention people here at all: people should NOT be tethered to one type of activity at one level of the Pyramid.

If we would put human forms to the Pyramid layers, the Factory would be the hands, doing the laborious work faithfully and efficiently. The Matrix would be the brain, doing smarts like gradient boosting random forests. And the Experience would be the personality: the easy, the flexible, the powerful, the fast, the visual… Every part of an app/solution/service, including UI, MidTier, Server, CI/CD, Technical Support…, contributes to the Experience.

All activities, regardless where they are in the pyramid, must be carried out in clear understanding of the context we’re in: the market, the competition, the academic researches, the open source innovations and the frontiers…

The Matrix

Analytics involves a vast variety of intertwined technologies of different natures (theoretical, computational) and may be applied to even more diverse set of domains, as long as they generate data. The successful application of analytics to domain problems hinges on ingenious combinations of domain knowledge and technological know-how. The Technologies × Domains Matrix is the core competency of any analytics shop.

And the matrix is where people really come into play. It can be considered as the third dimension of the Matrix: Talents. There are three types of experts that drive the data → information → knowledge → action value creation process.

• Technology Experts. All of them possess the intimate knowledge of a specific technology: Its underlying theory, practical considerations, its vendors and pros/cons of each, its communities… Many of them possess the engineering know-how to advance the state of the art of the technology by inventing and implementing more performant incarnations of the technology.
• Domain Experts. They possess the domain knowledge which dictates what technology to use, how to model the problem and how to efficiently reach the solution. True domain experts work outside of SAS for our customers. Internal domain experts mostly likely reside in customer facing divisions such as sales, technical support, education and Center of Excellence (COE).
• Solution Experts. Solution experts possess both technology know-how and domain knowledge, although on a lesser sophisticated level than technology and domain experts. Their value-add lies in critical thinking, problem solving and inter-personal skills picking brains of the technology and domain experts.

The Matrix is virtual in that technology, domain and solution experts are physically dispersed in different Pyramids best matched with their expertise yet collectively they function as a holistic body such as an organization wide open committee in the BlueFootprint system.

The Organizational and Architectural Layers

The Pyramid is an organizational stack where developers in lower layers provide value to developers in higher layers. Because the matrix manipulation library is built in the Factory layer, developers in the Matrix don’t have to build it. Because the Gradient Boosting Trees is built in the Matrix layer, developers in the Experience layer don’t have to build it. Because of externally available software as a services (SaaS) and open source software (OSS), internal developers don’t have to deal with infrastructure, and they can stand on shoulders of open source innovations.

The good old n-tiered architecture, on the other hand, is implementation details that ensure the most efficient flow of data across different layers of apps and solutions. The architectural layers are evolving as technology evolves. Who knows whether the mid-tier will still be there in five years!

Regardless how the organizational and architectural layers are actually implemented (matrix, architectural layers embedded in the Experience layer of the Pyramid…), both drives the business value for customers.

In a word, data is what flows through architectural layers while values are what flow through organizational layers, developer values to be specific.

The Democratization of Power

How to make people thrive in the inverted pyramid? Make them feel powerful! But we can only have that many managerial positions… Well, there are at least five different types of power in any organization and the positional power is the least influential kind.

Provide Mobility to Foster Referent Power

Referent power is a personal power that comes from being trusted and respected by others. If one is stuck at one place doing one thing, there’re only that many people around that she may draw referent power from. Allowing people to work at different places on different things with different people makes work more fun and people more connected.

Celebrate Expert Power

For nerds like me, expert power is the most achievable form of personal power. Don’t worry if you don’t have the charisma and interpersonal skills called for referent power, just be the best at what you’re doing, you’ve got expert power! Technology, domain and solution experts form the backbone of our competitiveness and they must be rewarded accordingly. Perks and visibilities are better directed to them rather than their know-nothing do-nothing people managers.

Distribute Positional Power

When the three types of formal power (legitimate power, reward power and coercive power) are concentrated in the hands of the position holders, bad things happen: nepotism, a** licking, credit taking, double standards, dead weights… Google the company tackles these organizational diseases by distributing positional power, especially the reward power, to committees of non-position holders. Take promotion for example, the managers have no say in a person’s promotion other than nominating references for or against the promotion seeker.

Everybody involved in the process get some taste of power: The employee doesn’t have to a** lick his manager to initiate the promotion process. The disappointed collaborator of the promotion seeker gets to speak the truth as a reference. The promotion committee members feel powerful as they execute their judge duties ……

With positional power removed or reduced, position holders now need to use their referent and expert power to legitimize their position:

• Managers Become Coaches. Coaches are experts themselves in an area of the Matrix. They own concrete, deliverable and whole (not only the juicy part of) things. 70% of their time they work like normal technology/domain experts do. 30% of the time they coach junior experts and provide support for senior experts.
• Directors Set Directions. Directors are experts in even bigger area of the Matrix. They still work, mostly in the Lab layer, tinkering with and thinking about technological directions in five to ten years’ time frame. They’re allowed to own only the design or prototype part of things.
• Presidents Preside. Presidents operate at strategy level where they exercise guidance, direction and control. When conflicting directions arise from different directors, they decide with authority. When wrong orders come down from executives, they disagree, persuade and commit.

Managers who don’t coach, directors who can’t see/set directions and presidents who lack the authority to preside have no place in The Pyramid. The lowest people have the sharpest eyes. Bad managers must be rooted out ASAP with the committee driven performance appraisal process (Yes, engineers get to sit in the committee and evaluate their Directors and VP’s performance!). Managers’ life is never too comfortable in the Pyramid. For example, there is the “Directors’ Cut” lunch-n-learn series where directors are forced to show and share the direction she sees in her area of the Matrix. The directors are put on the spot, no “delegation” allowed! Managers positions are meant for the motivated who want to lead in addition to work.

You may use a 200-point font size to occupy the whole PowerPoint slide with the single word “empower”, but these are some of the ways that truly empower people…

The Crackdown on “Managing” and Meetings

Time spent on things other than thinking and doing are time wasted, and we waste a ton of time in over-convoluted managing and unnecessary meetings.

Automate Mechanic “Managing”

Imagine the shock I got when one of my defects attracted a comment from a VP asking for the goal date. Apparently, the VP didn’t have the time to read and understand the defect, as my last comment clearly stated that the fix has been checked-in and I simply forgot to mark the defect as “Fixed”. The comment most likely was not typed in, rather a copy-n-paste message for all defects missing the goal date. Well, if you call this “managing”, I can develop a rule-based robot to automate the job away:

You see, the robot is considerate enough not to bomb everybody on the defect in the beginning, yet resolute enough to apply harsher and harsher interventions until the all-mighty goal-date is surrendered! Basically, all the frenzy around defect metrics can be automated in a similar fashion, saving time and efforts spent on needless explanations, assurances, scrum and scrum-of-scrum status meetings between individual contributors and their managers, between the managers and their managers, between people managers and project managers… While all these can be automated in a system like BlueFootprint whose dashboard provides better high-level picture for executives.

The VP’s diligence and track records command my deep respect yet I feel sorry for her, if she thinks what I think: the single-minded and super-heightened focus on defect metrics, which are not truthful indicators of quality (defect free doesn’t mean bug free) and certainly not an indicator of great analytical user experiences, deprives the VP what she truly wants to focus her energy on: putting in place software engineering processes and standards and preventing defects from creeping into products in the first place!

As another example, where face-to-face meetings can be automated away: My documentation writer Marie always exceeds expectation and documents my stuff perfectly, yet we never had to meet with each other in a meeting room. Why?

1. Marie writes her things down. (Duh, she is the documentation writer.)
2. Marie use a PDF commenting server to collect people’s comments automatically, eliminating the need for face-to-face meetings.

Crowd-sourcing Managerial Tasks with Self-Services

We’re all adults and we expect to be treated like adults. If my feature development work is delayed by two weeks which messes up my tester’s schedule, I could just go and talk to my tester or even better simply update my ETA in BlueFootprint and let the system do the “talking”. Why do I have to bubble up to my manager who then set up a meeting with my tester’s manager who then inform my tester of the delay? Managers should be bothered only when the individual contributors couldn’t handle an issue themselves. The self-service app facilitates and automates much of the run-of-the-mill managerial tasks so managers can focus on what they’re supposed to do: coaching.

Dude, the Light’s Up, Go Home…

The theater janitor woke me up and I was back in reality. The movie I saw was Joker, and I thoroughly appreciated all the metaphors and the tragedy there-in: isolated, bullied and disregarded, the Joker had to descend into crime to find power. “For my whole life, I didn’t know if I even really existed. But I do, …,people are starting to notice”. The city of Gotham needed a hero and it got a clown. For those who feel “we’re all clowns” like me occasionally at work, please don’t descend into slacking to get revenge. Rather, be the best at where you’re. No matter how niche the technology you’re tending is, and how much removed of your domain from the hyped, if you exercise your expert power, you are a star and people will take notice!

Appendix

The juxtaposition of positional and personal powers has several interesting duals, for example when it comes to performance: How much performance can be attributed to the position and how much to the person holding the position?

Positional and Personal Performance

If you manage a swim store in Florida and achieve twice the revenue of that of an average swim store nationwide, I’m not impressed, because the two times revenue is positional to Florida, and anybody put in the same position probably achieve the same thing. If you achieve 19 times of the national average by selling swim suits from Florida to Alaska via online channels, the performance is all yours.

I’ll refrain from naming the markets and domains whose performance is largely positional. The point is, don’t compare with other divisions internally, compare with competitors in the market. If your market share is shrinking year after year, even though you’re the number one internal revenue grosser, you’re still a loser.

Conditional Market Share

Accurate competitive market shares can be hard to get, well, talk to your internal sales guys! For all the customer biddings you participated, tally the total potential revenue and those bids we LOST to your competitors, you got your (conditional) market share:

Positional and Personal Admiration

Imagine everybody in an organization hang a camera on their neck and wear casual wherever they go because the previous CEO does so, and now every man wear pink, every man and woman play guitar from the book “Who Moved My Cheese” because the current CEO does so (what? It’s not a children’s song book?), anyway, that’s positional admiration. It’s in essence the blind worship of the position from either the brainless or the spineless driven by either ignorance or personal ambitions.

Personal admiration on the other hand admires the person responsibly as a whole package: her strengths and weaknesses, her accomplishments and failures, her visions and illusions… When marching order coming down from the person makes no sense, personal admirers disagree, try to persuade yet ultimately commit to it. Just like I rant here all day yet I exceed expectations in execution of whatever orders coming down from the command chain.

References

1. French and Raven’s five forms of power.
2. The (beautiful) pyramid was made with an Excel sheet, per this instruction. I hate self-promotion but this one deserves it in every way!
3. Google’s promotion process from one ex-Googler’s experience. It may sound like a rant from a disgruntled, but please focus on the disclosed promote process which is largely more fair.
4. A true rant from a former disgruntled SnapChat employee. But don’t read the post, read the Hacker News discussion which is full of insights into how Silicon Valley companies actually operate.
5. “Layers in software: data to value” mistakenly claims that the “value” layers replaced the “data” layers. The two layers are orthogonal, and the data layers will never go away (maybe compressed in the “Experiences” layer of the Pyramid).