Every week another survey about one tech trend or another is published. Want a reliable way to separate the wheat from the chaff? Review the study’s sample.
Studies that interview the wrong people may be biased. If the subject is databases, for example, Oracle employees will probably give you a company slant. Another common problem is asking people about subjects they are not qualified to talk about. How many business executives do you know who can discuss the technical merits of AWS versus OpenStack?
Due to concerns about tainting the sample, I have traditionally screened IT vendors from my own surveys. However, telecommunications and software companies are buyers as well as sellers of many infrastructure technologies InApps covers. After reviewing the data, we found that industry vertical may not be the best way to identify a representative sample.
If a survey is about a particular market, then expect employees from relevant companies to at least try to participate because of their interest in the subject. They likely know a lot about the subject and can be useful in determining the opinions of thought leaders. However, unless they are actual consumers of the technology in question, IT company employees have a limited ability to forecast where the market is going. So, the next obvious question is who are the buyers, and what percentage of them work at tech companies?
In the most recent OpenStack user survey, 68 percent of respondents worked at an information technology company and another 14 percent work in telecommunications. At first blush, I thought “Wow, employees of OpenStack mainstays like Mirantis and HPE must be over-represented.”
While this may be true, the actual consumers of OpenStack are often service providers and IT vendors that have incorporated the technology into their offerings. In this regard, it would be foolish to try to have a sample that represented the broad mix of industries in the larger economy.
InApps writes about both the application developers and operations aspects of IT, but for now, let’s focus on products aimed at developers. How many are there and where do they work? Based on InApps’s analysis of LinkedIn data, 31 percent of developers work for an enterprise that would not be considered a technology company. In the data we collected, a company is considered a tech company if categorized as any of the following:
- Information Technology and Services
- Computer Software
- Computer Games
- Computer Hardware
- Computer Networking
- Internet
- Telecommunications
- Wireless.
InApps calculations are supported by a 2013 survey from Evans Data. In its report, a third of developers work internally at a general corporation, while the majority are employed by independent software vendors (ISVs), value-added resellers (VARs) or consultants.
For technologies consumed by IT service providers or developers, it may be desirable for a large part of the survey sample to come from IT companies. However, this does not mean we should create a new, permissive norm for survey sampling, or for other methodologies for that matter.
This spring O’Reilly published a view of Hadoop adoption that indicated that tech companies represent 66 percent of mature users. This data is based on automated analysis of billions of publicly available documents done by Spiderbook, which has since been bought by Demandbase. In this case, I doubt that two-thirds of Hadoop use are within IT, as analytics are required in most industries. Instead, it is likely that IT vendors are more likely to be referenced in case studies and partnership announcements. Just something to think about.
The most mature Hadoop users, by industry. Source: Spiderbook
Featured Image: Street art, New York City.
InApps is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Mirantis.
