An Elastic technical review.PART 1 <<<1 - Overview2 - Elastic Overview 3 - Compare to MDBPART 24 - Strengths, in HaikuPART 35 - Use Cases6 - Final TakeawaysThis post is so long that it had to broken up into 3 parts. Use "View Whole Thread" to view it all.OVERVIEWAs I said before, EVERY COMPANY MUST be a tech-driven company. This new tech landscape is driving the hyper-growth stories we are seeking here, as companies sprout up to be the "picks and shovels" plays that are supplying this gold rush of technological innovation. They are creating new tools that are enabling companies to solve their own problems themselves, that are cross-applicable to every business, across every industry. And luckily for us, it still seems like the early innings! So I've had a series of posts, doing technical deep dives to try to isolate what companies are doing to make their service so sticky, as well as other posts about the tech behind companies products:* Insights from Elastic conference. https://boards.fool.com/insights-from-elastic-conference-340... * Cutting through the FUD on MDB. https://boards.fool.com/cutting-through-the-fud-on-mdb-34145...* An Okta technical review. https://boards.fool.com/an-okta-technical-review-34177580.as...* MDB goes mobile. https://boards.fool.com/mdb-goes-mobile-34194422.aspxI have spoken on MDB twice before, and haven't felt the need to dive very deep into the technical details of their product line, as it seems pretty easily understood -- once you know what a NoSQL document store is, you know what MongoDB excels at. But let's walk through their history a bit and where it has put them strategically. [Reminder, I call the company MDB to differentiate product from company. Elastic thankfully makes it easier so I refer to them by name. Downside is this means I use the word "elastic" over 200 times here.]MDB started by making an open-source NoSQL database, then it sold support and tooling for that database to enterprises that were using it for either their internal database or as an embedded database within their products. Once cloud computing took hold, MDB then started providing a managed, vendor-neutral, cloud hosting service for its core database... one that its customers flocked to, for its scale, high availability (HA), ease of use, and the fact it completely saves them money by eliminating costs around infrastructure and ongoing maintenance. MDB's approach has them creating a core platform around MongoDB of tools that reduce customer friction -- for either self-hosted or for managed Atlas. They have apps for data exploration (Compass) and a mgmt interface (Ops Manager or Cloud Manager). They have SaaS tooling around Atlas service, like a serverless platform (Stitch), a cloud migration tool, and a visualization dashboard tool (Charts, in beta). And as I discussed before, it's now increasing customer flexibility and increasing the applicability of its platform with its moves into being a synchronized mobile database (also in beta, but, finally, with a major acquisition to help them move faster).Elastic is an incredibly similar storyline to MongoDB -- the database and the company -- but their technology stack and solutions it provides and the TAM it has are a bit tougher to understand. So today, we dive deep into Elastic, and its suite of technologies that underpin its appeal to customers, and its new product lines spinning from that core. How do I know the company's products so well? Besides being a software developer that works with a lot of databases and data feeds, I have worked with Elasticsearch (not the full ELK stack) for the past 4 years, using it as a vital piece of my architecture. More recently, I have run some parts of my stack within the AWS environment the past year (more for the data storage resources than compute). I'm about to try using managed Elasticsearch in AWS, and, besides it being a data store within my stack, I'm also about to start using ELK for APM and monitoring of my stack. [No brainer that I should have implemented long ago, it's just I don't have the time; too many other interesting projects (around data streaming) to do!] Warning: There is a lot to like in Elastic, in ways that excite me beyond what MDB is doing. But for that, you'll need to do a lot of reading below to get to the Final Thoughts. But don't just jump there... I recommend the middle bits too! Dammit, don't miss the haiku! My last deep dive into the tech behind a company was Okta -- and this deep dive is even longer. For one, I know the company way better. For two, it was worth diving in deeper into their strengths and strategies.ELASTIC OVERVIEWElastic is known for its suite of products it calls the Elastic Stack. (It was first called the "ELK" stack after the first 3 products, and it's still mostly called that.) After starting as a company focused on its Elasticsearch search database, Elastic shifted gears early on towards being a solution for specific use cases, when it acquired companies with complimentary tools, then integrated them into a platform around its core engine.Their Getting Started docs describe the core well enough: Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements. https://www.elastic.co/guide/en/elasticsearch/reference/curr...Elastic Stack is:* Elasticsearch, the search and analytics database at the core. * Logstash, the data processing and transformation pipeline, for data ingestion into Elasticsearch. * Kibana, the visual interface over Elasticsearch, with data visualization dashboards and a cluster & data management interface.* Beats, light-weight data shippers utilized for transmitting monitoring data from network and systems, and ingesting them into Elasticsearch.* "Features" (formerly X-Pack), are modules that enhance the capabilities of Elastic Stack, such as adding cluster monitoring, alerting, data security, reporting, machine learning (ML), and a visual presentation app called Canvas. [Not sure why Elastic decided to now blandly call it "Features". I guess they let the new marketing intern have a go at it.]The Product Line:Elasticsearch (ES) is the open-source NoSQL database at the heart of the stack, which provides search and analytic capabilities over your data. It is built over the open-source Apache Lucene indexing engine (created by Doug Cutting, eventual creator of Hadoop), but with a focus on cluster capabilities to manage and search over ever-growing datasets. It is a distributed software, making the engine as powerful as the cluster hardware it is installed on, and the cluster can easily expand over time. A developer uses a REST API interface or native Java libraries to store JSON data, and can then search or analyze that data via the query interface. Bottom line - if you are slicing and dicing over large data (hundreds of Gb or more) or big data (hundreds of Tb or more) for search or analytical purposes, Elasticsearch is ideal. It is popular, having 40k stars on Github and a high DB Engine rating (#7 overall and #1 for search). There are a few competitors in the open-source space... but as I discuss later in detail, the real competition to Elastic is elsewhere. Alternate open-source search-based databases on the market are: * Apache Solr, which is also based on Lucene. But it pales in comparison, having only 2.5k vs 40k stars for ES on Github as a sign of its popularity, and #16 on DB Engine ratings. It has very little of the surrounding ecosystem of tools that ES has. It emerged out of CNET, and later was merged with the Apache Lucene project itself. An company, Lucidworks, was created to support it, and they created their own enterprise edition of Solr called Fusion. [Contrary to anecdotal commentary on a recent post, this is not a company I would look to as supplanting Elastic in any way. But yes, it is a direct competitor.]* Apache Druid, a OLAP/BI analytics engine, which has 8k stars on Github. It's also a clustered data engine but is a lot more convoluted & complex to run. It came out of work done at Metamarkets, an marketing analytical SaaS company, and now has an enterprise company, Imply, supporting it. Druid is a columnar store, which means it tracks field data together, not as separate rows. This allows for advanced analytics. [I'd keep my eye on it, but ultimately it isn't gaining much momentum, likely due to the complexity of the cluster setup required.]Other competition is other scalable NoSQL databases, like MongoDB or Cassandra. Developers may prefer and pick those, but the search and analytical capabilities of those pale in comparison to what Elasticsearch is capable of. Plus there is no tooling around those for doing ingest, like Logstash and Beats. Kibana is the visual interface over Elasticsearch. At it's core, it's a visualization dashboard web app that allows you to rapidly graph ad-hoc queries against your ES data, and create persistent visualization dashboards. It is pretty similar to another open-source visualization dashboard, Grafana, but is more closely tied to having ES as the underlying database, and, unlike Grafana, provides a mgmt interface over your ES cluster and its data. Kibana also includes "out-of-the-box" dashboards for specific server apps you are monitoring with Beats. As you hook up monitoring over your server-side apps (say a PostgreSQL database or an Nginx web server) with Filebeat and Metricbeat, you can use Kibana's dashboard templates honed for that specific application as a starting point, then customize it from there as desired. It has plug-in modules for time-series data visualization (Timelion), geospatial visualization over maps (Elastic Maps), and exposes dashboards over many of the X-Pack features like ML anomaly detection and APM monitoring. Logstash is the ingestion piece, that allows for continuously reading in logs from various servers, transforming log entries into JSON objects and ingest them into ES. It has a rich system of data pipeline steps, where you can convert, enrich, filter and transform log data prior to ingestion.Beats is a collection of light-weight "data shippers", each specific to collecting a type of data feed from remote servers or devices. This includes a log file shipper, metric shipper, network traffic monitoring and more. They are installed as system agents on your servers, which allow you to continuously collect data and ingest it into ES. Each beat comes with a wide variety of server apps it can work with out-of-the-box. As mentioned before, on the opposite side in Kibana, it includes sample dashboards for each server app that are honed to that app's metrics or logs.The Beat flavors are:* Filebeat for log file monitoring. It can tie into logs from server apps (like databases or web servers) for log monitoring across your infrastructure, and can do container log monitoring of Docker and Kubernetes.* Metricbeat for real-time server metric monitoring, including metrics from system (like cpu/disk/memory usage, or temperature readings) or server app (like databases or web servers). It can perform real-time metric monitoring of your stack, and can do container metric monitoring of Docker and Kubernetes. * Packetbeat for real-time network traffic & latency monitoring.* Auditbeat/Winlogbeat for system-level auditing of Linux/Windows systems.* Heartbeat for system uptime monitoring. Simpler & lighter uptime check than Metricbeat.* Functionbeat for real-time serverless function monitoring (i.e. monitor your AWS Lambda function).Elastic has built up a collection of CODE-FREE out-of-the-box solutions here within Beats and Kibana for monitoring use cases. Elastic has curated a list of log & metric Beats packs and sample Kibana dashboards for a wide-variety of popular server-side applications (like PostgreSQL, MySQL, MongoDB, Cassandra, Nginx, Redis, Kafka, Kubernetes, and even Elasticsearch itself). For each system within your architecture, like your PostgreSQL database, you can use Filebeat to pull in logs, and Metricbeat to pull in real-time metrics; then within Kibana, you can pull in the PostgreSQL-specific dashboard templates, so that you can start immediately visualizing those metrics and logs. From there, you can then customize the dashboards and the queries used on the visualizations and reports within, as desired. Beats, with its focused modules and "out-of-the-box" dashboards within Kibana, form a monitoring solution that has competition:* InfluxDB, a time-series database, competes on certain use cases like monitoring. It has a TICK stack that is similar to the ELK stack, with tools for ingest and metric shippers. For visualization you must rely on outside tools like Grafana. There is an enterprise company that built it is InfluxData, and they, of course, offer managed cloud hosting. I spoke with an Elastic VP at their ElasticON conference last year, and he didn't see customers picking InfluxDB over Elastic Stack. Perhaps a slanted view, as I do see InfluxDB in use at my work and in the field. * The open-source Prometheus monitoring platform is popular. It too must be paired with the Grafana visual dashboard board. Features/X-Pack are modules within the Elastic Stack to enhance the platform and help focus Elastic Stack over specific use cases, and to help manage your cluster. [I added marketing link to the higher impact ones.]Modules include:* Monitoring module for cluster monitoring.* Security module for role-based access security over your data, down to document/field level.* Alerting module for cluster alerting & data monitoring via queries (get notifications on spotted errors).* Reporting module to generate, schedule, email reports. Generate PDFs of your Kibana dashboards.* SQL module to allow for ES querying via well-known SQL data querying language. (It limits the search capabilities from what you have in the API, but is helpful for developers.)* Hadoop Connector to directly connect Elasticsearch directly to Hadoop for querying.* Graph tool to explore relationships in your data (use cases: fraud detection, security analysis, user recommendations). https://www.elastic.co/products/stack/graph* ML for performing machine learning over your data and visualizing results (use cases: detect anomalies, isolate patterns, pinpoint causes, demand forecasting). https://www.elastic.co/products/stack/machine-learning* Canvas app for creating presentation visualizations from real-time queries (aka over live data). Great for real-time demo presentations and interactive info-graphics.https://www.elastic.co/products/stack/canvasWith X-Pack, they used to all be Premium and only come with Enterprise Support tiers. However, back in April 2018 they changed their licensing. (MDB soon followed suit with their licensing change in October 2018.) They made a new pricing tier, Basic, that includes several modules that are now free. [More later on the licensing changes.]There are a few other ancillary services and products that Elastic has:* APM Server (server-side app) for collecting APM metrics from your application code and saving it to ES. Not officially a "Feature" module, as it is a stand-alone server-side app. Definitely not a Beat, either, as it requires code changes - you embed Elastic's SDK into your code so it can start streaming live metrics from your app to the APM Server. APM may allow you to bypass needing to pull in app logs via Filebeat - it's akin to hooking up Metricbeat directly into your app.* Elastic Map Service to provide high-quality global and regional maps, with which you can overlay geospatial data in Kibana. https://www.elastic.co/elastic-maps-service* Elastic Common Schema is an effort from Elastic to try to unify data schemas for common activities (logs, metrics, APM, networking data). It is an attempt to unify how to store like-data from different sources (say, Cisco's firewall logs vs Fortinet's). Elastic is hoping to convert various data sources into a single common schema, which allows you to simplify the search, analysis and visualization queries you do on that data.Getting Support:There are 4 tiers of support subscriptions. https://www.elastic.co/subscriptions- Open-Source (free) - includes ELK, APM Server, most of Beats, limited Elastic Map Service - Basic (free) - includes free X-Pack modules (monitoring, SQL), APM Server UI, and all of Beats and Elastic Map Service- Gold - provides biz hours support, plus includes rest of X-Pack (w/ only the basic Security, and no ML), and Elastic Monitoring service- Platinum - includes 24/7/365 support, plus all the above, adding full ML & full Security modules (SSO, ACLs down to document field, encryption at rest), cross-cluster replication. I griped above about how X-Pack is now just referred to as "Elastic Stack Features", a kind of bland label instead of a product name. But after reviewing that pricing chart showing features that turn on and off per tier, they are clearly blending in these modular features now; they are now directly embedded within their product lines, like ML and Security both being heavily integrated across all of Elastic Stack. I think they are losing "X-Pack" because they don't want to think of them as separate plug-ins -- they are integrated modules, and that has only expanded across other parts of the Stack. Kibana, Beats and Logstash have internal modules that turn off and on. This is why the open-source purists are upset -- there are proprietary modules embedded inside open-source packages. But, it seems Elastic is pretty clear about what is and is not included in each tier. Though its a bit too detailed, perhaps - they need some higher level overview of what modules are turning on and off. Support levels are spelled out clearly. You have to go Platinum tier for ML capabilities and advanced Security features. For Gold & Platinum support levels, they offer custom training and consulting services for additional fees.Hosting:As for hosting their stack, a customer has the typical options plus an extra one for running your own on-premise cloud:* Host it themselves (self-managed, self-hosted), either on-premise or in the cloud (on EC2 or Docker instances). It's a complicated software to configure, but obviously doable. I've used it entirely for free thus far -- but elsewhere in my company they just bought enterprise support to use Elastic Stack for monitoring infrastructure (system logs via Filebeat and metric collection via Metricbeat). It's up to the customer if they need support and the added features enabled by Gold/Platinum tiers.* Elastic Cloud is their hosting service, where Elastic can host and manage Elastic Stack clusters for you in your cloud-provider of choice. Elastic Cloud service came from their acquisition of Found in early 2015. MDB followed suit, and released Atlas service in mid-2016. Managed vendor-neutral cloud hosting is the big reason these companies are growing revenue so strongly.* Elastic has a 3rd option, Elastic Cloud Enterprise, which allows a company to deploy Elastic Cloud onto its own infrastructure via Docker, and use it as an internal, on-premise cloud where they can create and manage multiple Elastic Stack clusters.COMPARE TO MDBThere are many similarities between them ...+ Both are focused around a core open-source NoSQL document store, accessed via JSON-based REST APIs or native libraries. Both engines are cluster-able and can be horizontally scaled easily (by adding more nodes to the cluster). Both data engines are built on replicated shards, which enable high availability (HA), resiliency and scale.+ Both founders created companies around that open-source database engine that provided enterprise support and continued adding features, tools and eventually platforms around their core database. Both then expanded to create managed vendor-neutral cloud hosting of their data engine (MongoDB Atlas vs Elastic Cloud). One strength for both over the cloud vendors: the fact that the authors and maintainers of these complicated clustered database engines are the ones best suited to running a managed instance. Put the experts in charge!+ Both provide platforms containing tools around that core database. Both have apps for managing the cluster, data exploration and visualization. When you buy MDB Enterprise Advanced subscription, beyond enterprise support you get to use their mgmt interface app (Ops Manager or Cloud Manager) for monitoring and backup, advanced modules for security & analytics, data visualization tool (Compass), and also get a commercial license to embed Mongo in your released product. When you buy an Elastic Stack Enterprise subscription, you get enterprise support as well as expanded capabilities of X-Pack plugins for security and ML.Yet, some major differences ...- MongoDB is a general-purpose document store with a wide set of use cases. Elasticsearch, having much better search & analytical capabilities and more flexible scaling, is a specific-to-purpose document store with a narrower [but expanding] set of use cases. If you manage a collection of data objects that has infrequent search or analytics needs, you pick MDB. For data that you need to slice and dice continually with queries to search and analyze it, you pick Elastic. And for data that is "ever growing", you pick Elastic.- Given this more limited set of use cases, Elastic has had to fight harder. They have rapidly expanded their product line by acquisition, adding tools and services that helped build their core database into a platform. MDB is building its platform itself, and IMHO is subsequently moving way slower. Their new product Charts seems too little, too late -- there are many other viz dashboarding tools that do this already (from open-source Grafana to proprietary Tableau).- Both have an "Open Source" focus, and both try to address having cloud-providers become competitors, using their own database against them. However, they have different approaches on how to address their open-source licensing to combat competition. MDB is trying to prevent cloud-providers from running a hosted cloud service using MongoDB (making them direct competition against their own Atlas service), by changing the licensing on their core database (making the OSS purists angry). Elastic is keeping the core database fully open-source (Apache 2.0 license), but is changing the licensing and open-source strategy of their bundled 'mostly free' X-Pack modules (also making the OSS purists angry, but this time its about the fact it's bundled in open-source ELK). Elastic seems content to let cloud-vendors be competitors to Elastic Cloud, and letting their feature-rich modules be the differentiator. - Speaking of ecosystem, Elastic released a set of plug-ins for the ELK stack in 2016 that they called X-Pack, for non-core features like monitoring, security, alerting, and reporting. They started with all plug-ins being Premium (enterprise license required), but now source code is publicly available, but not open-source, and these modules are now free-to-use in their Basic tier. A subset of them are still premium and require an enterprise subscription. The Elastic Stack releases are bundling the open-source and Elastic licensed modules together. (Yes, this may cause some licensing confusion.) - While both MDB and Elastic have a managed cloud-hosting service plus provide users support for their self-hosted & self-managed databases, Elastic has a 3rd option - Elastic Cloud Enterprise (also from the Found acquisition). It allows their Elastic Cloud product to be installed on your on-premise infrastructure, so you can easily manage multiple ELK clusters on an internal cloud.- Elastic isn't building a cloud side and a on-prem side to their platform like MDB is. It's all Elastic Stack in the Elastic Cloud, just hosted at whatever cloud provider the customer desires, and managed by the finest experts one could find -- thems that wrote it! There isn't tooling appearing in Elastic Cloud that isn't in core platform, unlike MDB with their Stitch serverless platform. However, the downside is that their Elastic Stack releases must bundle the proprietary modules side-by-side with the open-source products.- One striking difference as I walked through the product line, is the number of use cases it solves that DO NOT INVOLVE CODE. MDB is for developers only, to embed into their application stack. Elastic is for that, but also for non-developers to use without needing any custom development. IT can hook up Beats for monitoring infrastructure or network traffic. Enterprise users can feed in datasets with Logstash, for staff to query, visualize, or apply ML in Kibana. I expect this trend to continue, as it really opens up the applicability as to who can use the product line.- Best of all, Elastic is making exciting moves that are moving their company beyond being a do-it-yourself tool provider. There is something afoot! [More on that soon under TAM section. Keep reading! But I'll give you a hint, it rhymes with "class".]To be continued, in Part 2 (due to TMF post size limit)...-mujilong ESTC (7%)
Muji,Not sure what it says about me that i open Sauls board and see the title of your post and got really excited!Pretty sure what it says about you is that these detailed writeups by an actual tech user are gold and greatly appreciated. Thanks!Dreamer
- I want to find the one executing better after a few Qs then move mostly to that one. Anyone in the collective want to start tracking and posting a head-to-head numbers comparison? (Please! I'm too busy pontificating here!)---Why not own both mdb and estc longterm, as they dont quite overlap?What if they are literally the #1 and #2 best stocks you could buy fornext few years. Just trying to understand why you are leanong towards either/or vs holding both?Thanks again...tremendously valuable posts!Dreamer
Wow. Thanks muji. That is an amazing post. You know more about Elastic than I guess I know about anything. In your first post you said: ...if you are slicing and dicing over large data (hundreds of Gb or more) or big data (hundreds of Tb or more) for search or analytical purposes, Elasticsearch is ideal. So what is your best guess of TAM? I am not a computer guy, so I have no idea. But it seems to me the number of people who would need to slice and dice hundreds of terabytes of data is not that large. Am I wrong?Thanks for taking the time to make this post.Jeb
And this is an investment board at its very best!Thank you muji!! What an incredible write-up. It simply amazes me that this sort of information is available to an ordinary guy like me.Thank you TMF and Saul for hosting this awesome board.
Awesome post muji!!! I’m a non-techie and have couple of questions. When comparing MBD to Elastic you stated multiple nodes needed for Elastic (min 3) and 0 for MDB? Does that make things more expensive and more complicated for Elastic VS MDB?What portion of the NoSql market assuming no other database company existed would you put MDB VS Elastic today and what trend do you expect to see a year or 2 out?With the amount of data getting larger every year would you not expect Elastic to grow faster in the near future than MDB?What do you think of the graph at the bottom of the page of the below link comparing MDB and Elastic? Will continual lower ram prices compensate for MDB speed in the long run VS Elastic. http://bitcom.systems/blog/moving-mongo-to-elasticsearch/Thanks.LafleurPS I have been following/lurking all you on Saul and NPI for years. I am indebted to you all. Saul, what you have created here is truly amazing! Thank you all.
Muji,Absolute dynamite!I think we’ll be seeing more from Elastic. The shares have the disadvantage of debuting so much higher than the IPO. But if the business maintains the momentum it has, the stock will have its day in the sun. Slowly accumulating.Darth
Muji:Wow....that was a very long and technical series of posts.....thanks for the effort!There a few negative issues with ESTC including:1) margin declining from 77% to 71% over past year and half.2) Revenue growth rate declining 3) Huge stock lockup right after this next earnings call...date of earnings not yet announced?4) AWS competitive open source environment (Open Distro)These items probably explain the stagnation of the stock the past couple months and the impact of AWS remains yet uncertain IMO......will they recruit some of the more prolific open sourcers to their camp??I agree with you that MDB and ESTC have somewhat different approaches to trying to protect their IP....but I am not convinced that ESTC’s model of trying to innovate their way to competitive advantage without licensing protection is more sustainably competitive than what MDB and other open source have been doing by trying to license their IP AND ALSO innovate.In fact, it will greatly surprise me if ESTC doesn’t follow the same playbook as MDB and Redis labs. IMO, we are more likely to see these open source model companies walk in step than to go separate business paths/models (witness the recent dual announcements of GOOG partnership for BOTH MDB and ESTC).But recognizing that ESTC has had the overhang of the above 4 items.....the top three likely go away assuming they continue to grow revenue at a breakneck speed.The larger more financial discussion is whether there is really an open source business model that can wildly succeed for a profit-based company......that has really been called into question of late with the growing near supreme power of the large cloud providers and AWS in particular.This article looks at this open source business model as it pertains to MDB but it really is about all open source models. It questions the viability of the open sourced for profit by parallels with the record industry:https://stratechery.com/2019/aws-mongodb-and-the-economic-re...Economic Realities and the FutureLittle of what I wrote is new to folks in the open source community: the debate over the impact of cloud services on open source has been a strident one for a while now. I think, though, that the debate gets sidetracked by (understandable) discussions about “fairness” and what AWS supposedly owes open source. Yes, companies like MongoDB Inc. and Redis Labs worked hard, and yes, AWS is largely built on open source, but the world is governed by economic realities, not subjective judgments of fairness.And that is why I started with music: it wasn’t necessarily “fair” that music industry sales plummeted, and yes, companies like Apple with its iPod business made billions off of piracy. The only reality that mattered, though, was that music itself, thanks to its infinite reproducibility, was as pure a commodity as there could be.The argument that he makes is that it really doesn’t matter how much better ESTC or MDB get (the innovation part)....the question remains as to what minimum performance is demanded/acceptable to the market to accomplish what just needs to be accomplished.Anyway, just food for thought for long-term holders of any of the open source companies and their business models. The Gorilla game of yesteryear started with a requirement for open “propriety” software as having leverage for the greatest moat......that is obviously not open source. MDB and Redis have tried to move closer to that modeling at least slightly using a new licensing model.....I expect ESTC still needs to do the same and stick together with MDB, work together and act together.These open source companies are threatened in very similar ways from AWS. Etc......they can learn from each other for the benefit of them all.
One important note is that Elastic and MDB aren't necessarily competitors. MDB is more of a database disruptor along the lines of the traditional SQL databases (SQL Server, Oracle, MySQL, PostgreSQL). In my opinion, its biggest threats in the NoSQL space are via Microsoft Azure's CosmosDB and Amazon's AWS DynamoDB. With more and more companies going to the cloud, they will be tempted to play in one of these areas instead. It is very interesting that MDB's Atlas provides the ability to choose where the DBs are housed (can choose to have it in AWS or Azure among others). NoSQL is great for loading lots of data and especially, data where the structures can change. That said, you hit upon the biggest issue with NoSQL databases; search. Elasticsearch is a searching technology. Think along the lines of when you type something into google and it auto completes; figures out what you want. Many companies who go NoSQL move the data to a big data type of repository for analysis and reporting purposes. In fact, even in SQL databases, this is the ideal step for analytics and AI. For companies with big search needs, loading the data to something like Elasticsearch or something similar (Solr, Azuresearch, AWS Cloudsearch) is ideal. Elastic might have designs on being both search and main data store but they need to resolve a lot of issues first (security, reliability). Lafleur, you asked about the nodes. That is gets into one of the big benefits to NoSQL. It is a much better data scaling mechanism called sharding. The data is horizontally scaled rather than the traditional vertical scaling by SQL databases. This makes it much cheaper because having multiple smaller computers is cheaper than one big beefy one. The world is evolving. The days of a company only having one database technology are gone or coming to a close. The cloud is making this easier for companies big and small.
Open source does not necessarily mean what you do will be done or can be done by everybody else. Does it? Ultimately one could say that whatever functionality implemented by a code could be done by anybody else. Once that beat or that melody is out there one could use it to create something else.From the past, Red Hat has also touted open source. You don’t think they have had a business?If you argue that the issue is them being ‘open source’ and you look at the stock movement then why is Mongo going up and Elastic not in recent months?What do you think will happen after the lock-up? tj
Thanks all for the responses and questions. Feel free to hit the smiley face by my name to "Favorite Fool" me, I kinda like this trophy-chalice icon that has appeared next to my name recently. Here are some answers to the questions up-thread from Dreamer, Jebbo, Lafluer, Duma and apwickman26.-----------Dreamer:Why not own both mdb and estc longterm, as they dont quite overlap? What if they are literally the #1 and #2 best stocks you could buy for next few years. Just trying to understand why you are leaning towards either/or vs holding both?I probably didn't put it very elegantly - holding both is exactly my plan, ultimately in roughly equal allocations. MDB has been on a huge tear, so clearly wins on relative strength. ESTC is likely still held back by share lock-up so I (and many others posting portfolios here, including Saul, Gaucho, Fleiberman, and Bear) have been slow to build up a position more than a starter 2-3%. After this deep dive, I was impressed enough to start increasing my allocation now, rather than wait for lockup to pass. What I meant in my closing statements was that we should be able to have a good view into their execution & strategies by comparing their trajectories from here, so if one falters we can act appropriately. I really like basket situations like this, like buying both SHOP and SQ and watching them both succeed. I argued against selling MDB in late Feb, due to AWS and Lyft news (it then popped at earnings), and bought more Okta early April after my review -- both of those decisions have turned out VERY nicely. I hope the same for ESTC. I nibbled back in the mid-60s back in Nov, needing to research them more. It has been very range-bound the last 2 months since peaking in late Feb. Once past the lockups and having another Q or two of stellar numbers, and I think we'll be sitting pretty.--------------Jebbo: So what is your best guess of TAM? I am not a computer guy, so I have no idea. But it seems to me the number of people who would need to slice and dice hundreds of terabytes of data is not that large. More companies than you'd think. Having to churn through terabytes of data is why Hadoop got created. I'd say all the Fortune 2000 and then many IoT or analytics oriented companies have datasets that size. Elasticsearch is built for large datasets, but any data that starts to outgrow the native search capabilities where it is stored can leverage it (say, anything more than 20 million rows or over 20Gb, and it may start exceeding the limits of the database it is in). Since I just had to look it up for work, I've got 33Tb in 300B documents in my 6 node Elasticsearch cluster, using it as a time-series database, with metrics going back to 2015.Guess of TAM? No idea, I just pontificated on the business lines and strategy. I feel that search is relevant to nearly every business w/ an app or website w/ any kind of data (blogs, articles, comments), much less all the trove of specialized datasets out there.While MDB has a lot of applicability to modern apps, where I like ESTC's moves are:- Elastic Stack is relevant to non-developers, for things like monitoring infrastructure or network traffic.- Elastic's moves into SaaS services to provide solutions for those non-technically inclined companies that don't want to run their own database.-----------Lafleur: When comparing MBD to Elastic you stated multiple nodes needed for Elastic (min 3) and 0 for MDB? Does that make things more expensive and more complicated for Elastic VS MDB?Yes. Which is more a reason to use managed hosting. Plus you can size it to your current needs and easily scale as needed from there.With the amount of data getting larger every year would you not expect Elastic to grow faster in the near future than MDB?Well that is an interesting thought. I'd say any company using either platform can be expected to have ever growing data. But given that Elastic Stack has tools to auto-ingest logs/metrics via Beats, that I would side with it being a greater velocity for them.What do you think of the graph at the bottom of the page of the below link comparing MDB and Elastic? Will continual lower ram prices compensate for MDB speed in the long run VS Elastic. It's an interesting chart. But their solution is vertically scaling by increasing RAM - there is only so far you can go. It's pretty easy to see how MDB performs on full-text search if it can keep the dataset in memory (and when it can't, it hits a wall where it starts getting super slow). They make no mention of setups (cluster size, etc) they are comparing so it's all pretty ignorable. -----------Duma: but I am not convinced that ESTC’s model of trying to innovate their way to competitive advantage without licensing protection is more sustainably competitive than what MDB and other open source have been doing by trying to license their IP AND ALSO innovate.THEY DO HAVE LICENSING PROTECTION. The choice they are making is in protecting modules within their surrounding eco-system by changing the licensing on the periphery, instead of changing the licensing of the core. The end goal is the same - new features that differentiate them from AWS's offerings are protected. In fact, it will greatly surprise me if ESTC doesn’t follow the same playbook as MDB and Redis labs. IMO, we are more likely to see these open source model companies walk in step than to go separate business paths/models (witness the recent dual announcements of GOOG partnership for BOTH MDB and ESTC).It's very clear that is not happening, so expect to be surprised (though how one is surprised by an eventual lack of an action, I'm not sure... I guess I'll just come back in a year and yell SURPRISE at you). Those who decided to take the periphery tack (Elastic and Confluent are the 2 I know of) have been very vocal from the start about their decision & the reasoning behind it, in blog posts from the CEO. They are trying to be very open and deferential to the open-source community about this necessary shift that the enterprise companies behind open-source software is need to take cloud providers if they hope to stay relevant in hosting it.There is a lot of debate about this in the OSS world. Changing the core is rightfully seen as extremely drastic, as it is much more susceptible to unforeseen complications. Any easy one for forsee, is that many companies are extremely careful about reviewing licensing of their selected tools, and may have picked MongoDB because of its original permissive Apache 2.0 license. If they have doubts, or even legal reservations, about the new license, they are COMPLETELY FROZEN at the last version under Apache 2.0 (v3.6). Way to screw them over! At a minimum, it requires legal review at a lot of companies using MongoDB internally or embedded within their software product or SaaS service.Redis and MongoDB's choice of changing core license is getting wayyyy more push-back than the periphery choice of having proprietary modules & tools around the open-source core. Redis had to go back and clarify their license a 2nd time, it was getting so much heat. MDB is got some serious side eye from Red Hat, who removed their software from their repos, and MDB had to eventually pull their new SSPL license from consideration at OSI. See this mailing list post from MDB CTO whining about it (http://lists.opensource.org/pipermail/license-review_lists.o...). In looking at the note it was replying to, it's clear what the push back is about -- the wording about preventing SaaS services "substantially similar to that of the Program". So it appears that SaaS companies that use MongoDB under the hood as a major component in their service have MAJOR CONCERNS about this new SSPL license. Changing the license on the core open-source database is having the rug pulled out from under you. It requires a legal review at a minimum, but these companies could be extremely upset about what is essentially a broken promise by MDB -- expectations were set in stone that MDB decided to change for THEIR BENEFIT ONLY. Maybe MongoDB feels that this shouldn't scare SaaS companies, as they are a general-purpose document store, and SaaS companies likely are using it for a purpose that isn't directly competing against MDB. If Elastic adopted this license on core Elasticsearch, they'd would be directly stifling companies like Swiftype, which built SaaS services over Elasticsearch that are "substantially similar to that of the Program" -- which Elastic acquired! I'm guessing they may want more of those types of acquisitions. And, funny thing, SSPL license is full-on attack against services like mLab... which they happened to acquire right before that licensing change. You mentioned the dual announcement from GOOG, but that has everything to do with GOOG wanting media attention around its choice to partner with these companies, and not end-run around them like AWS; it is NOT about MDB and ESTC being partners or being in lock-step.These open source companies are threatened in very similar ways from AWS. Etc......they can learn from each other for the benefit of them all.I agree they can learn from each other. One side is going to learn that the other made better strategic choices. I tried to be more neutral about it in my write up (like "who knows which is better"), but I think the signs are shouting pretty loudly about which choice is better.MDB and Redis have tried to move closer to that modeling at least slightly using a new licensing model.....I expect ESTC still needs to do the same and stick together with MDB, work together and act together.Redis is really in the weakest position here - their tool is easiest to replicate, and there aren't major enhancements being made to it from here. MDB had little choice in the matter, as they don't have the platform of proprietary tools to make the periphery play (until they had Stitch, anyway - that's a pretty big differentiator). Elastic and Confluent have an ecosystem of tools that were already proprietary, just with more permissive licenses. They are buttoning that up real quick, as those tools are differentiators against the core offering. -----------apwickman26: One important note is that Elastic and MDB aren't necessarily competitors. I never stated they were. I thought I made clear the difference that Elastic is a search-oriented database, MDB is general-purpose document store for data collections. I am comparing them not as direct competitors, but as very similarly structured businesses with similar history -- creating and maintaining an open-source NoSQL database and now having success at providing managed cloud hosting of that database. MDB and ESTC are both in the hypergrowth Saul universe, so I think it is important to view them in tandem - either to compare their execution and strategies to evaluate them, or to invest in both as a "sticky dev tooling w/ cloud hosting" basket.MDB is more of a database disruptor along the lines of the traditional SQL databases (SQL Server, Oracle, MySQL, PostgreSQL). In my opinion, its biggest threats in the NoSQL space are via Microsoft Azure's CosmosDB and Amazon's AWS DynamoDB.Yes. I didn't rehash that as I already covered all that earlier in my "Cutting Through the FUD on MDB" post. https://boards.fool.com/cutting-through-the-fud-on-mdb-34145...For companies with big search needs, loading the data to something like Elasticsearch or something similar (Solr, Azuresearch, AWS Cloudsearch) is ideal. Thanks for bringing that up - I should have mentioned Azure Search and AWS Cloudsearch. They've been around a long while as cloud-based competitors to Elasticsearch, but don't compare at all to Elastic Stack's popularity. We don't have numbers from cloud providers on usage, but those two vendor-specific databases are way down the list on DB Engine rankings (at 54 for Azure and a way pitiful 94 for AWS). So to copy a bit from my MDB post linked above:REGARDLESS of these competing products, Elasticsearch is the clear king of search databases.REGARDLESS of these competing products that are more "native" to their respective clouds, Elastic has thrived, in part by building an entire eco-system of tools (Kibana, Beats) & modules (APM, ML) that those products are missing. REGARDLESS of many cloud providers providing managed Elasticsearch themselves, Elastic has thrived as a cloud-neutral provider of managed hosting.Elastic might have designs on being both search and main data store but they need to resolve a lot of issues first (security, reliability).Elasticsearch is NOT angling to be your main data store. It's not striving to be the "single source of truth", as we call it in the biz (aka being the sole database for all of a company's data). The common pattern using it is that you are pulling pieces from various "sources of truth" into Elasticsearch (say, your HR records but not payroll tx), with it acting as a storage engine over it. I have to dismiss the issues you list. Elastic HAS solved security, it's been a proprietary module in X-Pack for years, where you can control security down to the field level. It's the first thing AWS had to address with Open Distro (they added a security module). As for reliability, stability has been a major focus since v1.5 (now on v7.0), and Elastic has been very diligent over the past several years about shoring up bugs and weak spots of their platform -- writing distributed software over a cluster is extremely difficult. And with shard replication, it has been geared for high-availability from the start.- mujilong ESTC
Muji,Thanks for your detailed posts. I am invested in Elastic. I see that you use Elastic quite a bit. I am curious what would prompt you to become a paying customer. I realize if you want features like security, alert, and monitoring or their SAAS products or service you have to pay. But curious if most like you can get by fine with the free version thus limiting their TAM.
Found a blog post that references Muji's fantastic overview :SNIP... ESTC is moving in the SaaS direction though. With a series of acquisitions, ESTC is well positioned to shift away from selling infrastructure software/source code and toward being an enterprise application vendor. App Search Service (powered by Swiftype which they acquired), Site Search Service, and the new Enterprise Search Service – these are all closer to what I would consider SaaS products. ESTC also just acquired Endpoint, and will use it to security application space. CMF_muji correctly points out that this “true” SaaS approach is an upside option for ESTC. I think it’s more than that. It’s a way out of this whole “existential threat to business model” quandary. This is a step function change in business model – selling application instead of selling enabling infrastructure. Your value proposition is different. The customer’s mindset is different. The SaaS user will focus on fulfilling the business case and ignore the infrastructure. They will not play around with Github and source codes. At that point the whole debate of open source versus propriety becomes moot. That is a promising thought, but one that is a long way off. The value proposition, customer set, as well as ESTC’s sales and marketing approach all have to change. This is by no means an easy move. https://greytabinvest.blogspot.com/2019/06/elastic-estc-saas...
Thanks for the blog.To cut the chase, the key issue is here:CMF_muji correctly points out that this “true” SaaS approach is an upside option for ESTC. I think it’s more than that. It’s a way out of this whole “existential threat to business model” quandary. This is a step function change in business model – selling application instead of selling enabling infrastructure. Your value proposition is different. The customer’s mindset is different. The SaaS user will focus on fulfilling the business case and ignore the infrastructure. They will not play around with Github and source codes. At that point the whole debate of open source versus propriety becomes moot. That is a promising thought, but one that is a long way off. The value proposition, customer set, as well as ESTC’s sales and marketing approach all have to change. This is by no means an easy moveMuch more complexity than a MDB...so it seems.I have yet to see anyone, including myself, be able to establish theTAM for its various niche business plays......”optionality” maybe......but for what calculated TAM???
Best Of |
Favorites & Replies |
Start a New Board |
My Fool |