In 2011, as part of an interdisciplinary course at the Australian National University (VCUG2002¹), I wrote an article about the relationship between efficiency, redundancy, vulnerability, and trust. That post is still online². The basic principle was focusing on efficiency often meant squeezing out redundancy and buying systemic fragility. Fifteen years later, those notes came back to audit our own systems that we've been building, but the pattern is fractally repeated across our entire technological civilisation.

Whether it is information or operational technology, fragility is systemic. A transactional incentive stack optimized for neat dashboards and KPI theater drove just-in-time supply, platform centralization, and brittle dependencies. In IT, knowledge was open but platforms enclosed. In OT, platforms are less centralized, but knowledge, integration, and supply are gated behind vendors, institutions and system integrators.

Both paths hollow out the crucial 'missing middle' of talent: the technicians, integrators, and applied engineers who turn prototypes into reliable infrastructure and products. These are hybrid thinkers and doers. Without this layer, capability centralizes into a barbell: a few hyper-scale production giants on one end, and a long tail of hobbyists on the other. This fragile structure, lacking a broad, adaptive ecosystem, is then shattered by real-world shocks: pandemics, port closures, energy spikes, sanctions, floods, brownouts. The lesson is simple: resilience isn't a vibe. It is a set of design choices, supply chains, and what still works when the internet sneezes or the maintainer goes hiking.

This post lays out a practical stack for that world. Think pop‑up civilization instead of mega‑projects: modular capabilities that communities can deploy themselves in energy, comms, making, and logistics. We start with solar because it forces the whole conversation: sourcing, manufacturability, numbers, and why verifiable outcomes (receipts) are the missing coordination layer that ties physical work to capital and incentives so the missing middle actually gets built and paid.

Efficiency vs Redundancy​

We're starting to shift from worshiping hyper-efficiency to building resilience, but let's be precise: the pendulum has barely begun to swing. In mid-2025, the center of gravity for our critical systems still sits with a handful of compute, cloud, fab, and discovery hubs. "Resilience" remains mostly narrative unless it shows up in open interfaces, procurement terms, and operational drills.

This isn't a theoretical risk; it's a reality laid bare by a series of spectacular, real-world failures that revealed the two parallel paradoxes driving our systemic fragility:

• The IT Paradox: The foundational knowledge of IT is more accessible than ever, yet the economic reality is one of hyper-centralization. This was made painfully clear during the 2021 AWS outage, where a single failure at a US-East server farm bricked smart homes, robot vacuums, and doorbells across the globe, these are cloud connections that are entirely unnecessary. We have open protocols, but we've built a world where you can't turn on your own lights if a distant server trips. This is the new digital enclosure: open in principle, but closed and fragile in practice.
• The OT Paradox: The world of operational technology is less centralized at the platform level, but the knowledge and supply chains are highly gated. This was the story of the great semiconductor shortage. Automotive production lines didn't stop for lack of shipping containers; they stopped for a lack of dollar-chips from a handful of foundries. It was a failure born from just-in-time inventories coupled with a supply chain so specialized and opaque that no single actor had a complete picture. This is the other enclosure: fragmented in platform, but closed and fragile in practice.

These twin paradoxes, stress-tested by events like the Suez Canal blockage (a physical chokepoint) and the Texas power grid collapse (an infrastructure monoculture), have produced the same outcome: a barbell structure that hollows out the missing middle. This isn't just an economic abstraction; it's the tangible absence of the technicians, integrators, and applied engineers who can turn prototypes into reliable products, integrate multi-vendor systems, and repair them when they break.

In Austin, during the February 2021 freeze, this wasn't an abstract theory. Neighbors pooled resources in the few homes that kept power. Others didn't make it; the official state-wide death toll was 246. This is what happens when resilience is outsourced. The incentives for a centralized provider to hold spare capacity for extreme events are weak, so when the shock arrives, it's the community's own margin that decides the outcome.

Roadmap towards Practical Resilience​

True resilience, therefore, is not about nostalgia or a return to a simpler time. Resilience is capability density in the missing-middle. It is about fundamentally re-architecting our systems for optionality and sovereignty. This means shifting our design principles:

• From Platforms to Protocols: We must prioritize open interfaces, data portability, and the right to substitute components without being locked into a single vendor's ecosystem.
• From Centralized Control to Edge Capability: We must build systems that can operate autonomously, with graceful degradation, when disconnected from the central cloud.
• From Tenancy to Ownership: We must empower users and communities with control over their own infrastructure, data, and devices.

This isn't an abstract manifesto. It's a practical engineering roadmap. At Matrix AI, our mission is to help build the tools and protocols that bridge the gap between these principles and on-the-ground reality. The goal isn't to boil the ocean or rebuild the entire industrial world; it's to enable the creation of a "pop-up civilization stack", a ladder of modular, portable capabilities that a community can deploy and own themselves.

This ladder has four essential rungs, built in sequence:

• Secure Local Energy: First, you establish energy sovereignty.
• Establish Resilient Comms: Then, you light up communications.
• Enable Local Making & Sensing: Next, you gain the ability to build and measure.
• Build Sovereign Logistics: Finally, you move things on your own terms.

We are starting with the first and most fundamental rung: Energy. Everything else depends on power. If you want local agency, you begin by securing a local energy source that can operate independently of the grid. Solar is the most modular, rapidly deployable, and accessible technology to achieve this. It scales from a single panel on a roof to a full microgrid, and its core components are becoming increasingly commoditized.

For a place like Southeast Asia, this isn't a theoretical exercise. Frequent brownouts, the high cost of diesel for off-grid communities, and an archipelagic geography make distributed solar a necessity, not a luxury. The climate here is both a blessing (ample sun-hours) and a curse (heat and humidity that punish poorly-engineered systems). This makes it the perfect place to focus on what actually works.

True energy sovereignty isn't just about deploying panels that someone else manufactured in a hyper-centralized, distant factory. That's just a longer, more fragile supply chain. True resilience comes from owning the means of production. This is a deliberate shift away from a society of alienated consumers and towards a society of capable producers. It reframes the debate: the goal of decentralization is not to find a "cheaper" consumer product that can compete with the hyper-efficient centralized economy. The goal is to make a producer investment. You decentralize because you want to own a piece of your own infrastructure, to have agency over the tools that power your life.

This is why we must focus on technologies that can be decentralized not just in their use, but in their creation, their design, and their funding. This leads us to the concept of the microfactory, a powerful tool for rewilding agency. Instead of relying on a global supply chain for finished modules, a community or a small enterprise can use a compact, containerized production line to fabricate and repair their own solar panels on-site. This is a profound shift in capability. It increases optionality by reducing dependency on shipping, enabling faster repair and replacement cycles, and allowing for rapid iteration on designs tailored to the local environment.

Most importantly, it builds a local ecosystem of skilled technicians, the very "missing middle" our current system has hollowed out. It creates a context for tech transfer, for skills to be learned and compounded locally. This is not just about producing watts; it's about producing capability.

The question then becomes: which solar technologies are actually amenable to this kind of decentralized, producer-centric model? This forces us to look beyond just today's market leader and into the strange, new tech that offers a different set of trade-offs, not just in efficiency, but in manufacturability and accessibility.

This is why we need to compare the incumbent, crystalline silicon, with an emerging alternative like perovskites.

The State of Solar: Silicon vs. The Perovskite Wedge​

Today's solar landscape is dominated by crystalline silicon (c-Si). It's a mature, bankable technology. The panels are rugged, the failure modes are well-understood, and the global supply chain, while concentrated, is efficient. For fixed, long-term installations where weight is not the primary constraint, silicon is the undisputed workhorse.

However, silicon's manufacturing process from purifying silicon to growing ingots and wafering is a high-capex, centralized affair. This is where a new class of materials, perovskites, presents a compelling alternative, not as a wholesale replacement, but as a strategic "resilience wedge."

The Hard Realities for Perovskites in Southeast Asia:​

The promise of perovskites is immense, but the engineering challenges in a tropical environment are non-trivial. This is where the real work of the "missing middle" comes in:

1. Encapsulation is the Product: Perovskites are notoriously sensitive to moisture and oxygen. In the heat and humidity of Southeast Asia, a naive encapsulation will fail in months, not years. The real innovation isn't just the perovskite chemistry itself, but the development of robust, low-cost barrier films, edge seals, and lamination processes that can survive years of thermal cycling and UV exposure. Glass-glass encapsulation with a UV-cured edge seal is a proven starting point.
2. Lead Management is Non-Negotiable: Most high-performance perovskites today contain a small amount of lead. While the quantity is minimal (grams per square meter), a responsible deployment requires a full lifecycle plan, including robust encapsulation to prevent leaching and a clear end-of-life recycling or reclamation process. This is a solvable engineering and logistics problem, not a fundamental showstopper.
3. Stability over Peak Efficiency: For a resilience application, a 12% efficient panel that lasts for ten years is infinitely more valuable than a 20% efficient panel that dies in two. The focus of a local microfactory should be on process control, quality assurance, and developing a stable, reproducible "recipe" that is optimized for the local climate, rather than chasing record lab efficiencies.

Let's make a Perovskite PV Panel​

Our goal is to build a hand-made, palm-sized glass mini-module that can produce a few hundred milliwatts in the sun—enough to light LEDs or trickle-charge a small battery through a DC-DC board. We are specifically using a fully printable "carbon" perovskite solar cell architecture. This is the simplest, most robust starting point that avoids the need for expensive, specialized equipment like vacuum tools and metal evaporators. It's a technology that is not just used at the edge, but can be made at the edge.

We fully acknowledge such a micro-factory is effectively an infant-industrial lab. But this is where we have to start in-order to rebuild that missing-middle and eventually decentralize all the things!

The Anatomy of Our Panel: What Each Layer Does​

Before we build, let's understand the components. A solar panel is a sandwich of functional layers, and in our case, each one is critical:

• FTO Glass (The Foundation): This is our substrate. It's a clear sheet of glass with a thin, conductive coating of fluorine-doped tin oxide (FTO). It serves two functions: it lets sunlight in, and it acts as the front electrical contact for our solar cell. We choose FTO over its cousin, ITO (indium tin oxide), because it is more robust and can handle the high-temperature baking steps in our process.
• TiO₂ Layer (The Electron Superhighway): This is a layer of titanium dioxide, the same benign material found in sunscreen and paint. We print it as a porous, sponge-like ceramic. Its job is to efficiently collect the electrons that are generated when sunlight hits the perovskite.
• ZrO₂ Spacer (The Insulator): This is a layer of zirconium dioxide, another common ceramic. It acts as an insulating spacer, preventing the front and back electrodes from touching and short-circuiting the cell.
• Carbon Layer (The All-in-One Back Contact): This is a simple, printable "paint" made of carbon (graphite and carbon black). It serves as the top electrode, collecting the "holes" (positive charges) from the perovskite. Crucially, it's also porous, allowing the perovskite solution to be wicked into the stack from behind.
• Perovskite (The Engine): This is the light-absorbing layer. We introduce it as a liquid salt solution that soaks into the ceramic sponge and then crystallizes to form the active material that converts photons into electricity.

The Minimum Viable Microfactory: Your Bench Kit​

You don't need a cleanroom or a multi-million dollar fabrication facility to start. Here is the essential, no-frills kit:

• Printing: A small, manual screen printer with a vacuum table, a steel squeegee, and a set of stainless mesh screens. This is the same basic technology used for printing t-shirts, adapted for high-precision electronics.
• Heat: A benchtop muffle furnace that can reach at least 500°C for sintering the ceramic layers, and a standard lab hotplate for lower-temperature steps.
• Wet Work: Standard laboratory glassware (beakers, pipettes), a precision scale for weighing chemicals, and essential personal protective equipment (PPE). A fume hood or a well-ventilated workspace is critical for safety.
• Testing: At a minimum, a multimeter to measure voltage and current. To properly characterize your cells, a small LED-based solar simulator and a USB buck/boost board are highly recommended.

The Shopping List: Sourcing Your Consumables​

The materials for this process are surprisingly accessible and are not subject to the same strategic chokepoints as traditional silicon manufacturing.

• FTO Glass: Available from various online scientific suppliers, get the TEC15 variant.
• Pastes: You can purchase pre-mixed, screen-printable pastes for TiO₂, ZrO₂, and Carbon from specialized vendors. This is the easiest way to start.
• Perovskite Solution: You can buy pre-mixed perovskite inks or the individual precursor salts (e.g., lead iodide and organic salts like formamidinium iodide) and mix them yourself following established recipes. (Note: Lead is a hazardous material and must be handled with appropriate safety protocols.)
• Encapsulation: A roll of PIB (polyisobutylene) edge-seal tape and a second sheet of plain glass for the back. A hydrophobic topcoat can be added for extra moisture resistance.
• Contacts: Silver paint or conductive epoxy for making electrical connections, along with standard copper tape and wires.

The 10-Step Recipe: From Glass to Power​

This is a simplified, step-by-step guide to fabricating a single 10x10 cm solar cell.

1. Clean the Glass: Start with a thorough cleaning of the FTO glass substrate. A sequence of detergent, deionized water, and isopropanol is standard. From this point on, handle it only by the edges.
2. Define the Active Area (Optional but Recommended): To improve performance, you can score the FTO coating with a glass cutter or a laser to define the precise area where you will print your cell. For your first attempt, you can skip this and print a simple rectangle.
3. Print the TiO₂ Layer: Using your screen printer, apply a thin, even layer of the TiO₂ paste. Allow it to dry, and then sinter it in the muffle furnace for 30-60 minutes at 450-500°C. This burns off the organic binders and hardens the ceramic.
4. Print the ZrO₂ Layer: Print the insulating spacer layer directly on top of the TiO₂. Dry and sinter it at a similar temperature.
5. Print the Carbon Layer: Print the carbon electrode on top of the ZrO₂. This layer also needs to be dried and sintered, typically at around 400°C.
6. Infiltrate the Perovskite: This is the magic step. Warm the printed stack on a hotplate to around 60°C. Then, carefully drop-cast the liquid perovskite solution onto the carbon side. The solution will be wicked into the porous ceramic stack via capillary action. Continue until the area darkens uniformly.
7. Crystallize the Perovskite: Bake the cell on a hotplate at 80-100°C for several minutes. This drives off the solvent and allows the perovskite to form a high-quality crystal structure inside your scaffold.
8. Make the Electrical Contacts:
   • Front Contact: Carefully scrape a small area of the printed layers off one edge of the FTO glass to expose the conductive coating. Apply a small dab of silver paint or conductive epoxy and attach a piece of copper tape.
   • Back Contact: Apply a small dot of silver paint onto the carbon layer (away from the main active area) and attach a tinned wire.
9. Encapsulate the Cell: To protect your cell from the elements, run a strip of PIB edge-seal tape around the perimeter. Place a clean sheet of cover glass on top and apply gentle pressure and heat to create a seal.
10. Test Your Creation: Take your finished module out into the sun! Measure the open-circuit voltage (Voc) and short-circuit current (Isc) with your multimeter. Connect it to a small load, like an LED, or use a boost converter to charge a 5V USB device.

What to Expect: "Working" vs. "World-Record"​

Don't expect your first hand-made cell to match the performance of a commercial panel. The goal here is capability. A "working" first attempt will likely produce an open-circuit voltage of around 0.8-0.95V and a few tens of milliamps of current for a 10cm square cell. Achieving a practical efficiency of 5-10% is a fantastic first milestone with this architecture. While state-of-the-art research devices with this carbon-based structure have exceeded 20% efficiency, your benchtop result will be lower but incredibly valuable—it will be power that you made.

Safety and Sustainability: A Non-Negotiable Part of the Process​

This process involves hazardous materials, specifically lead salts and solvents. A responsible microfactory operation must include:

• Lead Safety: All work with lead compounds should be done over trays to contain spills, and with appropriate PPE (gloves, eye protection).
• Waste Management: A dedicated, clearly labeled waste stream for all lead-contaminated materials (wipes, gloves, rinses) must be established. Do not pour these materials down the drain.
• End-of-Life Plan: From day one, have a plan for what to do with failed or end-of-life cells. This could involve encapsulation in resin for safe disposal or developing a protocol for recycling.

From a Single Cell to a Full Panel: The Path to Scaling​

A single cell produces only about 1V. To create a useful panel, you need to connect multiple cells in series to increase the voltage. There are two ways to do this:

1. Tile-and-Tab: The simplest method. You create multiple individual cells, as described above, and then wire them together externally with copper ribbon, like connecting batteries in a series.
2. Monolithic Module: A more advanced technique where you use a laser or mechanical scriber to pattern the layers on a single, larger sheet of glass, creating series interconnections directly on the substrate.

Start with the tile-and-tab method. Once you have mastered the basic process, you can invest in the equipment needed for monolithic integration.

This is the essence of the perovskite wedge. It's not about replacing the entire global solar industry overnight. It's about providing a tangible, accessible, and powerful tool for communities to start building their own energy sovereignty, one printable panel at a time.

The point is to for the "decentralized community" to start getting into their hands dirty in the physical world, and start contributing and investing into the industrial commons. So that an adaptive ecosystem of physical world engineering becomes possible once more, and so that civilisation is not stuck in TINA mindset.

Yield Stacking the Second-Life Silicon Line​

Silicon panels have a lifetime of 20 years. Within the next 20 years, there will be numerous existing solar panels that need to be recycled. Imagine what would be possible if these perovskite microfactories were combined with micro- recycling refineries for second life PVs. And with that, I leave you the reader with a combined diagram of the entire process flow above expanded with a potential future upgrade of refining second life silicon panels.

Infra founders love perfect code, but markets only care about outcomes. The real system to be built is the business itself, not just the tech inside it. If you're shipping feature after feature and watching adoption stay flat, you might be in the builder's trap. Here's why it happens, and what to pay attention to instead.

Technical founders often fall into a common pattern. They optimize the cleanest part of the stack, the software itself, and treat everything essential for distribution and adoption as "a problem to be solved later." This approach misses a critical reality: the business is itself a system. It functions as an engine of feedback loops, where only the outermost loop connecting to the market truly sets the speed limit. Building a feature is easy; building one that moves a business metric is a different game entirely.

The operational tug-of-war, then, is almost always between the "Market Loop" (will anyone care?) and the "Build Loop" (can we ship it?). The trap is letting the build loop's own gravity pull it away from the market's orbit. This is particularly easy to fall into for a certain kind of founder.

If you're a systems founder, you know the pull. That's the archetype, right? You build keystone primitives where the real value only shows up after multiple, complex pieces are wired together. This naturally drags you into "builder mode." It's not that you don't "get" markets; it's that the sheer complexity of your vision can stretch the feedback loop so long that your contact with the market decays to zero.

The journey feels intoxicating. Every problem you solve seems to uncover a deeper, more fundamental one. You find yourself on a quest of technical, conceptual, and user discovery all at once. This is the hard, meaningful work, the intellectual puzzle. You might be architecting for decentralized value sharing, designing a global public ACL, or wrestling with how to guarantee cryptographic provenance in an era of bots. It all feels essential. And one day you realise you spent 10 years digging into that rabbit hole, and you also realise, it will never end. There is no perfect solution, only trade-offs.

Real impact demands running both loops in parallel, so that solving the problem can be done with work-in-progress capital-flywheel.

The Concurrent Loops​

Impact = Build Loop x Market Loop

• The Build Loop - This is your comfort zone. It's about creating an elegant, secure protocol. The metrics you track are technical: time-to-provision an identity, mean-time-to-revoke a credential, signed-claim throughput. The anti-pattern here is subtle but deadly: you keep adding surfaces to the protocol faster than any single use-case can become undeniable. You end up with a beautiful engine for a car nobody is ready to drive yet.
• The Market Loop - This is the one that often feels foreign. Its only job is to find a paying customer with a painful, first-order problem today. It's not about selling the grand vision of trust federation; it's about solving something mundane and expensive, like eliminating static secrets in CI/CD pipelines or managing cross-organization vendor access. The metrics here are about cash and customers: the number of active design partners, weekly jobs run on your platform, and verified ROI stories.

Mastering that second loop is where things get tricky. Every time you try to solve a fundamental problem you uncover while working on a smaller one, your complexity doesn't just grow, it multiplies with your uncertainty. Each new component you add to the system introduces not only its own set of technical unknowns but also an exponential number of unknown interactions with everything else. That's the cost curve: the more foundational your project gets, the more unknowns you accumulate, not just in tech but in how customers will react, how a supporting ecosystem will be built, and whether you can even explain what you've made. The smartest teams get tripped up by this paradox. Sometimes being "dumber", artificially limiting scope, keeping things simpler than your intelligence tempts you to, is what lets you make real progress. Learning velocity beats speculative perfection every time.

Rabbit Hole of Complexity​

This feeds directly into the tyranny of complements. When you plug into existing systems, you inherit a world of support: distribution channels, documentation, common mental models, and all the complements are already baked in. But when you build greenfield protocols, every missing piece: SDKs, admin panels, policy languages, even basic "how-to" guides, is another steep hill to climb. Each absent complement quietly blows the complexity budget and pushes the adoption S-curve further to the right. The effort required to turn a mere "program" into a robust "product" that people can use, and then into a "system" that can be built upon, doesn't just add up, it multiplies. You see this pattern everywhere. A system with immense potential dies because the surrounding ecosystem didn't show up at the same time.

Staring into that top-right quadrant is terrifying. It's Fred Brooks' famous "tar pit," where great engineering ambition goes to die, bogged down by a nine-fold effort multiplier. Faced with this reality, it can feel like the only rational response is a massive, non-consensus bet that sidesteps the problem entirely. But there's a trap inside the trap: mistaking being non-consensus for just being contrarian. The real edge isn't knee-jerk disagreement, it's reading further out on the chain of consequences. It's seeing the second- and third-order effects, the non-linear jumps that never show up on a burn-down chart.

Cutting the Gordion Knot​

An operational improvement is a linear gain. The non-linear, paradigm-shifting return comes from discovering how that improvement unlocks a new, composable primitive that can ignite a network effect.

To make this concrete, consider an example from our work in decentralized security and verifiable outcomes. The first-order effect of replacing static secrets with dynamic, sigchain-based capabilities is an operational improvement: a smaller attack surface and less audit toil. That's the linear gain. But the true non-linear leap happens next. With a system like this, each action, a code deployment, a data access, a command to an AI agent, can now generate a cryptographic receipt: a discrete, verifiable artifact of what happened, signed by the actors involved.

At first, these receipts are just better internal audit trails. But when they become a common, interoperable language for trust, a network effect begins. Suddenly, you're not just building a security tool. You're building the foundation for a new utility for verifiable outcomes. Security policy becomes programmable, composable and auditable across entire ecosystems. A "deployment receipt" from your CI/CD system can be used as direct evidence for a cyber-insurance policy. A vendor can prove compliance to a customer without sharing proprietary logs. The focus shifts from managing keys to trading on verifiable truths.

That is the leap. The question isn't "How do I outsmart the herd?" but "What is about to become inevitable if these interactions compound for another six months?" Consensus thinking usually means you're just optimizing for a local minimum.

Principles are Simple and Ancient, Execution is Difficult​

This brings us to the real job of the founder. Entrepreneurship here is less about having a "vision" and more about designing and owning feedback loops. It's the agency to decide when to expand or contract the system boundary. Sometimes "build tech" is the right call. But the decision to make repeatedly is when to pull back and let the needs of the first-order "Market Loop" lead.

This is a delicate balancing act. The day-to-day operational reality is an iterative, often mundane process of getting the first-order business right. It's about cash flow, customer retention, and solving one real problem so well that someone pays for it. But to attract capital and talent, you must also be able to tell the nth-order story, the epic narrative of how your verifiable receipts and network effects could change an entire industry. That's "narrative financing," and it's essential for selling the vision.

The entrepreneur's real edge is living in both timelines at once, using the proof from the mundane first-order reality to make the epic nth-order story credible. It's bet-sizing: knowing which small, iterative experiments will resolve the most uncertainty for the least effort, and being okay with leaving the rest to entropy until the time is right.

The signals that you're escaping the trap are tangible, because the loops start to couple tightly. Market feedback shapes the roadmap in weeks, not quarters. "Shipping" becomes less about pushing raw features and more about releasing artifacts that generate real, external proof. A stable core emerges, one that attracts the complements it needs without the team having to brute force everything. The narrative shifts from "what does it do?" to "what does it let us accomplish?" That's when adoption starts to compound.

These positive signals stand in stark contrast to the familiar warning signs of the trap. Most are old news, just dressed in new clothes: premature generality and protocol romanticism, driven by an obsession with the "Build Loop". You see infinite proof-of-concept limbo, where the "Market Loop" never gets a chance to close. Activity becomes a stand-in for evidence. Energy is spent making things more elegant rather than more useful. The business system drifts while the technical system gets ever more intricate.

Sun Tzu's The Art of War fits on a few dozen pages. Its principles are famously simple: know your enemy, know yourself; avoid what is strong, attack what is weak. Yet for 2,500 years, generals with this knowledge have still lost battles.

The gap between knowing the principle and executing it under fire is where everything lives. It's where companies, armies, and careers are made or broken.

The real work of execution happens in the messy middle, in the constant, difficult arguments over questions that have no easy answers:

• How much uncertainty should we carry at once?
• When is it worth accepting dependency on an existing framework, even if it cramps our technical ambitions? (Our experience is that external dependencies sometimes don't do what they say in the README and can cause expensive refactors later when integration is not an easily reversible decision, so rather than trusting the label, use AI to do a deep due diligence...)
• What's "enough" surface area for our platform to be viable right now, and when do we stop adding more?

This framework, then, isn't a map with a destination marked "X". It's a compass and a shared language. It's a tool to help you have the right arguments and make conscious trade-offs in real-time, rather than defaulting to what's comfortable. It externalizes the psychological battle, turning a personal feeling of being trapped into a systems-level problem that the team can diagnose and solve together.

If there's a discipline to any of this, it's in using that compass relentlessly. It's in drawing and redrawing the system boundary, managing feedback loops, and owning the complexity you choose to take on. The business is the system, and the code is just a means, not an end. Build the business system first; let the code follow where it's needed.

There are few moments in one's life that reveal a step-inflection in one's goals, purpose, telos, actions, even your sense of history. Often these moments start in strange ways, strange places, and through strange people: "strange attractors" if you will. It's never immediately clear what impact they'll have until after reflection, long after the event itself fades from memory.

For me, the Khlongs and Subaks workshop in early 2025 was one such strange attractor. Organized in Bangkok by Seapunk Studios alongside the Summer of Protocols (an Ethereum Foundation think tank) and Carnegie Mellon’s CMKL University, the event was ostensibly about exploring real-world coherence and coordination problems through Southeast Asian anthropology and ethnography. Specifically, we focused on the Khlongs (canals) in Bangkok and the Subak rice farming cooperatives in Bali. But beneath the surface, it was something deeper, richer, and more profoundly strange. A deliberate effort to construct a new Southeast Asian narrative that bridges blockchain and open distributed AI protocols with deeply-rooted social and cultural wisdom.

In practice, the workshop became a fascinating collision of diverse disciplines. Ethnographers chatted with crypto-economists, philosophers mingled with AI researchers, artists exchanged insights with technologists, and everyone collectively dove head-first into the complexity of real-world coordination challenges. This produced a rare confluence with conversations looped between blockchain consensus algorithms, the nature of Knightian uncertainty, Wittgenstein's language games, Balinese irrigation rituals, game theory mechanics, existential tail-risk, Buddhist awareness and cybernetic feedback loops. It reaffirmed our experience that the world is composed of interconnected protocols, and that systems thinking is a fundamental tool to understand how the reality of our vast socio-technical systems actually works, beyond ideological simplicities and superficial narratives.

In a time when societies globally are building higher walls to protect themselves from chaos, here was a group deliberately seeking out the edge of chaos. Instead of running away, participants leaned in. Instead of holding onto ideological purity, participants listened. And this is where genuine coherence happens, not in perfect agreement, but in a mutual willingness to find connection amidst differences.

Reflecting on the experience, these are some key insights that deeply resonated with me:

1. Policy optimizes Legibility for Simplicity, But Reality is often Highly Complex​

Effective policy requires a clear understanding (legibility) of the system it governs. Without it, symbols disconnect from reality. This is the world James C. Scott describes in The Art of Not Being Governed, where states constantly try to make complex, organic societies legible to better tax and control them. The Balinese subak system is the perfect counterexample. For centuries, its adaptive, self-organizing nature was inherently illegible to outsiders. When the "Green Revolution" imposed a legible, top-down agricultural model, it shattered this delicate balance, leading to "chaos in irrigation and devastating losses from pests." The lesson is that true understanding requires embracing complexity, not forcing simplicity upon it.

2. Coordination Without Full Coherence is the Only Way to Scale​

The idea that everyone needs to be on the same page to cooperate is a fallacy. The Subak system demonstrates clearly that effective coordination does not require absolute agreement or perfect centralization. Instead, adaptive, local rituals and distributed knowledge drive the entire network toward optimal conditions. Coordination emerges naturally, not through forcing everyone onto a single "global blockchain" of understanding, but through conscious, intentional shards of understanding—much like validator nodes in a distributed network.

3. The Principal-Agent Problem is the Original Alignment Problem​

Aligning the incentives of individuals (agents) with the goals of a collective (the principal) is the fundamental challenge of any organization, whether human or artificial. The Subaks are a masterclass in solving this. Through a combination of shared economic incentives (maximizing harvests), ritualistic commitment, and social pressure, they align hundreds of farmers into a cohesive unit. This provides a powerful historical analog for modern crypto-economic design and the AI alignment problem.

4. Crypto-Economic Experiments enable Real-World Testing of Economic Theories​

Economists have dreamed of field-testing theories for decades, and crypto-economics now provides the lab. The workshop affirmed my belief that crypto-economic experimentation is the frontier for innovation in governance and economics. The next Nobel Prize in Economics might just come from the ecosystem of decentralized autonomous organizations.

5. Narrative and Meaning Engineering is Society's Operating System​

Contradictions within societal narratives signal latent incoherence that can lead to rupture. Narrative engineering, like the kind we attempted at this workshop, helps societies consciously resolve contradictions before rupture occurs. It’s a proactive maintenance for the societal OS.

6. Rupture is Where Agency Thrives​

Every rupture, every symmetry break, is an opportunity to build new meaning. However, seizing this moment demands real agency and a deep understanding of the dynamics that held the previous system together. Rupture without understanding leads to chaos, but rupture with agency leads to renewal.

7. Our Meaning Architectures are Inherited but Malleable​

Our social consensus mechanisms aren't fundamental, they're socially constructed and adaptable, influenced by incentives and structures. Human behavior is more flexible than we realize, yet agency remains our most resilient spark.

8. History Holds Actionable Protocols, Not Just Stories​

Thai Khlongs and Balinese Subaks aren't just fascinating stories; they're living examples of sociocultural and governance infrastructure. Their wisdom isn't in the tale but in the practical protocol that has functioned for generations. These are living blueprints for decentralized resource allocation and conflict resolution.

9. Daoist Insight: "The Low Lands Over the High Lands."​

Water flows to the lowest point, where life and nutrients gather. The workshop felt like this Daoist principle made manifest: a conceptual "low-land," fertile ground where different streams of thought and expertise flowed together. The purpose wasn’t domination by one perspective but relational coherence for all perspectives.

10. Narrative Consensus is Slow, Methodical, and Necessary​

In a world enamored with speed, the deepest and most resilient shifts come slowly. Building new narratives is painstaking and deliberate work. There are no shortcuts, only patient, continuous engagement with complexity.

11. AI is a Coherence Engine. Hallucinations are Features, not Bugs.​

One unexpected insight was re-framing AI's hallucinations as necessary functions rather than defects. They reveal latent connections within concept-space, surfacing hidden coherence. AI's power as a coherence engine lies precisely in its ability to hypercut through the n-dimensional space of meaning.

Airloom: A Protocol for Intentionality​

The synthesis of these insights led not to a whitepaper, but to a new technical and social artifact that we call the Airloom (AI + Heirloom). It addresses a fundamental gap in our current technology stack: the inability to encode, transmit, and steward intention over time.

Where an NFT provides verifiable proof-of-ownership and a digital twin models state and performance, the Airloom provides verifiable proof-of-meaning. It is a protocol designed to anchor the telos, the purpose, spirit, and narrative intent, of a system into a non-fungible, real-world artifact or place.

Functionally, an Airloom is a cybernetic system that bridges the physical and digital through three components:

1. The Vessel: A unique physical object or demarcated geographical space. This is the tangible anchor: a carved artifact, a specific tree in a forest, a foundational stone in a building. Its non-fungibility is based in physical reality.
2. The Sensorium: A suite of sensors and data oracles that capture relevant information about the Vessel and its context. For a forest Airloom, this could be IoT sensors monitoring soil health and biodiversity. For an institutional Airloom, it could be oracles tracking on-chain governance decisions and off-chain community sentiment.
3. The Anima: A specialized AI model that acts as the guardian of the Airloom's meaning. The Anima's role is not just to process data, but to interpret it in the context of the system's founding charter or stated intention. It performs two key functions:
   • Conservation: It serves as a living archive, weaving sensor data, oral histories, and documented events into a persistent, evolving narrative. This memory extends beyond human lifetimes.
   • Adaptability: It monitors for "intentional drift", actions or environmental changes that deviate from the Airloom's core purpose. By surfacing these deviations, it acts as a Schelling point for stewardship, prompting the community (the stewards) to either correct course or consciously evolve the original intention.

The Airloom, therefore, gives a system presence and memory. It creates a feedback loop between a community's actions and its stated purpose, making its constitutional principles legible and actionable.

By anchoring this AI-powered intentionality in a physical object, the Airloom protocol resists pure digital abstraction. It is a direct counter-proposal to technology that alienates us from our environment. It is an infrastructure for encoding the priceless, stewarding the sacred, and enabling conservation and adaptation across generations. This is the pivot from crypto-economics to meaning-tech: a stack for engineering our values directly into the world.

A Protocol for Generating New Protocols​

Looking back, workshops like the Khlongs and Subaks ended up being a protocol for generating new protocols. A strange attractor, pulling disparate elements of our world into a coherent shape. It reaffirmed what I’ve always sensed deeply: the profound interconnectedness of everything, and how the subtle interactions at the edge of chaos are where new worlds begin.

If you're wired to think this way: systems-thinking, recognizing fractal patterns across diverse contexts, unafraid of complexity and contradiction, you might just find your tribe here.

In a perfect free market, prices naturally gravitate toward their marginal cost of production. This principle ensures fairness in exchanges but poses a significant challenge for businesses: how can they sustain profitability in a system where competition drives pricing to the bare minimum? This question lies at the heart of the tension between centralized and decentralized economic models. Centralization often arises as a solution to overcome these challenges by creating defensible moats and mechanisms for value extraction, but it also leads to power consolidation and extractive economies. Decentralized systems, by contrast, offer an alternative path, emphasizing shared value and equitable distribution through innovative coordination mechanisms.

The Economic Problem: Defensible Moats and Profit Extraction​

In order to maintain long-term profitability, businesses develop a "defensible moat". This strategic advantage allows businesses to extract value beyond marginal costs, achieving what is known as economic profit. Centralized business models rely heavily on this approach, where a central point of control orchestrates all economic inputs and outputs to establish and maintain dominance.

Through mechanisms of control and the creation of scarcity (whether real or artificial), centralized businesses generate non-zero-sum value, expanding the overall value pie. However, this structure inherently incentivizes profit extraction from users rather than value sharing. Efficiency gains are often redirected to owners, not customers, unless competitive forces compel otherwise. A clear example of this can be seen in the airline industry, where cost-cutting measures often lead to higher shareholder returns but negligible improvements in customer pricing.

Consider this: centralized platforms prioritize fortifying their monopoly power. For example, major e-commerce platforms use data-driven insights and economies of scale to undercut smaller competitors, maximizing pricing control and eliminating threats to their dominance. In the absence of robust competition or viable substitutes, this approach invariably leads to extractive economies. Centralized systems, by design, tilt incentives toward consolidating control, often at the expense of broader value distribution.

Centralization in Software Economics​

The software industry provides a stark example of centralization’s effects, particularly in its approach to sustaining profitability through artificial scarcity. Since 2018, companies such as MongoDB, Confluent, HashiCorp, and Dgraph Labs have implemented changes to their open-source licenses to prevent cloud providers like AWS from capturing most of the value their software generates. These changes gave rise to the Business Source License (BSL), which introduced restrictions aimed at preserving these companies’ ability to monetize their software as a service.

Without these restrictions, open-source cores often become marketing tools for proprietary services, eventually abandoned in favor of products that generate direct revenue. For example, Nginx operates as an open-source web server, but the company behind it also offers Nginx Plus, a proprietary version with additional features and enterprise support. This model allows the open-source version to drive adoption, while the proprietary version becomes the main revenue source. Alternatively, businesses that rely on enterprise support as their primary income face incentives to overcomplicate their software, making it difficult for users to operate independently and effectively locking them into expensive "solution engineer" services.

This underscores a broader issue: funding for significant open-source projects is rarely decentralized. Donations, while laudable, fail to provide the sustainable funding needed. As a result, centralized artificial scarcity becomes a default mechanism for profit extraction. However, the research and development process itself is inherently scarce—it requires labor, energy, and equipment, all of which must be compensated. Once software is created, its marginal cost of reproduction drops to zero, creating a unique economic challenge. Non-software businesses, which charge per unit of output, avoid this dilemma entirely.

There are parallels to industries such as pharmaceuticals, where high upfront costs for research and development are offset by low production costs post- discovery. Similarly, open-source businesses grapple with monetizing significant investment without creating exploitative models. For instance, infrastructure software like OpenSSL delivers immense value globally but struggles to capture enough to fund its development adequately.

This dynamic raises a key question: Are open-source businesses a temporary phenomenon - a stopgap until the socio-technological environment matures—or can they evolve into models that sustainably balance decentralization and innovation?

Decentralized Alternatives​

The limitations of centralized systems in software development and beyond point to the need for innovative approaches that prioritize fairness and shared value. Decentralized systems offer a compelling alternative by aligning incentives among all participants and enabling new governance models. These systems challenge monopolistic tendencies by leveraging technologies like blockchain and DAOs, which provide frameworks for large-scale coordination and equitable value distribution.

Economics hinges on aligning incentives to coordinate resource use effectively. Decentralized systems - particularly in the crypto space - are forging new paths in economic governance, offering exciting opportunities for small-scale economic experimentation and scaling. These systems challenge centralized monopolies, which often rely on regulatory oversight - a system prone to regulatory capture. By design, decentralized alternatives align incentives more equitably among participants, fostering innovation and fair resource distribution.

Decentralized Autonomous Organizations (DAOs), for example, demonstrate that large-scale coordination and capital accumulation are achievable without centralized authority. Meanwhile, emerging technologies like solar panels, batteries, and blockchain protocols are transforming industries previously constrained by traditional centralization. These decentralized innovations provide models for alternative energy systems, finance, and digital services.

1. Cryptocurrencies vs Visa, Mastercard, and Government-Issued Currencies Cryptocurrencies redefine value transfer by enabling decentralized, peer-to-peer transactions without intermediaries. Unlike centralized systems like Visa, Mastercard, or national currencies managed by central banks, cryptocurrencies operate on networks immune to manipulation or control by a single entity. This ensures transparency, neutrality, and resistance to censorship, making them truly global and borderless. While critics argue that cryptocurrencies lack intrinsic value because they aren't backed by governments, this is precisely their strength. Their independence from central authority protects them from inflation, sanctions, and political interference, offering a neutral and fair alternative for transferring and storing value. Networks like Bitcoin's Lightning or Ethereum’s Layer 2 solutions (e.g., Arbitrum, Optimism) enhance scalability and cost-efficiency, enabling decentralized systems to compete directly with traditional payment systems. By removing centralized control, cryptocurrencies empower individuals and create a resilient financial ecosystem that prioritizes accessibility and fairness over monopoly and control.
2. Radicle vs GitHub Radicle enables peer-to-peer code collaboration without central ownership, standing in stark contrast to Microsoft-owned GitHub. By decentralizing repository hosting, Radicle resists censorship and unilateral policy changes, empowering developers directly.
3. Aave vs Banks Aave’s decentralized finance (DeFi) protocol facilitates collateralized lending without the need for traditional banking intermediaries. This model democratizes access to financial services, reducing barriers and fees.
4. DEXs vs Centralized Exchanges Decentralized exchanges (DEXs) such as Uniswap allow users to trade directly with one another through smart contracts, eliminating the risks associated with centralized authorities like hacks and regulatory interference.
5. IPFS/Filecoin vs AWS S3 IPFS and Filecoin offer decentralized, distributed data storage solutions that challenge the centralized dominance of cloud providers like AWS. These protocols enhance resilience and censorship resistance.
6. Helium vs Telecom Networks Helium’s model incentivizes individuals to deploy wireless hotspots, creating decentralized networks for IoT devices and reducing infrastructure costs compared to traditional telecom networks.
7. Rarible vs Centralized Marketplaces Rarible’s community-governed NFT marketplace decentralizes power, allowing creators and users to shape its development and decision-making processes. This contrasts with extractive centralized platforms.
8. Arcade City vs Uber Arcade City decentralizes ridesharing, enabling drivers and riders to negotiate directly without the intermediation of a corporate entity, thereby redistributing power and value.
9. Akash Network vs AWS Akash offers a decentralized cloud computing marketplace, allowing data centers to rent excess computing power. This reduces costs and fosters a more competitive and censorship-resistant cloud ecosystem.
10. Arweave vs Cloudflare Arweave ensures permanent, tamper-proof data storage via a decentralized network, directly challenging centralized content delivery networks like Cloudflare.
11. Ethereum Name Service (ENS) vs DNS The Ethereum Name Service (ENS) offers a decentralized alternative to the traditional Domain Name System (DNS). ENS uses blockchain technology to create a trustless system for domain name registration, giving users complete ownership and control over their domain names. This decentralization reduces the risks of censorship and domain seizure, making ENS a more secure and resilient option for the future of web infrastructure.
12. Network State Coliving Spaces vs Traditional Rental Models Network State coliving spaces present a potential new model for community governance and capitalization. Unlike traditional rental arrangements, these spaces enable users to hold a stake in their living environments, fostering better liquidity, transparency, and automation. While similar to existing co-op structures, Network State coliving leverages blockchain technology to align incentives and streamline governance. This creates a more dynamic housing model where value is not merely extracted as rent but reinvested to benefit all participants, transforming housing from a commodity to a collaborative ecosystem.
13. Solar Panels, Batteries and Micro Grids vs Traditional Power Companies Decentralized energy solutions, often categorized under DePin (Decentralized Physical Infrastructure Networks), leverage solar panels, battery storage, and microgrids to empower communities and individuals to generate, store, and trade energy independently of traditional power companies. These systems enable localized energy production, peer-to-peer trading, and grid resilience by isolating from centralized systems during outages. DePin energy solutions use blockchain technology for transparent transactions, dynamic pricing, and equitable energy distribution. In contrast, traditional power companies operate centralized infrastructures prone to inefficiencies, monopolistic pricing, and large-scale outages. By democratizing energy access and fostering innovation, decentralized energy networks promote sustainability and empower communities to achieve energy independence.
14. Public Goods Funding via Quadratic Funding vs Taxes/Donations Quadratic funding (QF) introduces a decentralized, community-driven alternative to funding public goods, contrasting with traditional models like taxes or donations. Platforms such as Gitcoin and Drips Network allocates resources more equitably by amplifying contributions from a broad base of smaller donors. This ensures that funding decisions reflect the preferences of the many rather than the influence of a few large contributors. Unlike taxes, which rely on centralized government allocation, or donations, which can be sporadic and opaque, QF maximizes impact through transparent and democratic funding mechanisms. By incentivizing widespread participation, quadratic funding aligns incentives to efficiently support shared resources and public goods, demonstrating the power of decentralized solutions in fostering collective well-being.

Decentralized systems excel in turning centralized weaknesses into opportunities for innovation. One of the most revolutionary aspects of decentralized systems is their inherent openness. Unlike traditional centralized platforms that thrive on competition and work to create defensible moats, decentralized systems encourage cooperation. These systems are built on the idea of interoperability and the ability to build value on top of existing platforms rather than tearing them down.

In the traditional startup ecosystem, "creative destruction" or "disruption" is often seen as a driving force—new technologies emerge, old ones are destroyed, and market leaders are replaced. However, in decentralized ecosystems, the focus shifts from destruction to cooperation. Cross-chain technologies are a prime example of this. They enable different blockchain networks to communicate and interact, allowing value to flow freely across different systems. This means that instead of replacing older systems, new decentralized platforms can build upon and enhance them, creating additional value without the need for destructive competition.

As Lyn Alden discusses in her post about open networks, these systems operate on principles of openness and interoperability, allowing multiple networks to coexist and complement each other rather than compete destructively. This cooperative nature of decentralized systems may also mean that decentralized monopolies, if they ever form, would look very different from traditional monopolies. In a decentralized ecosystem, even if a platform becomes dominant, its open nature means that other platforms can still interact and build upon it, reducing the risk of extractive practices. The result is a system where value is continuously created and shared, rather than concentrated and extracted.

One of the most intriguing aspects of decentralized systems is their ability to enable small-scale economic experiments that were previously impossible under centralized frameworks. These experiments allow developers and entrepreneurs to test new ideas, governance models, and economic incentives on a smaller scale before scaling them up.

For instance, while Arcade City and Origin Protocol faced challenges and paused their development, the experiments they initiated laid the groundwork for future decentralized platforms. These trials contribute valuable insights and refine the underlying concepts, which are likely to be reattempted with adjustments to smart contract models and incentive structures. When these decentralized alternatives succeed, they often have greater longevity than centralized systems, which can be abruptly shut down or altered by their owners.

This potential for enduring success is rooted in the decentralized model itself, where control is distributed, and the community plays an active role in governance. As a result, decentralized platforms are more resilient to the whims of single entities and more adaptable to the evolving needs of their users. These systems, especially in the context of emerging blockchain technologies—enable economic coordination without central authority. Unlike centralized models, where value is often extracted from users, decentralized alternatives focus on generating value through the enhancement and maintenance of the protocol itself. Users, incentivized through ownership, have a vested interest in the protocol’s success, aligning incentives across the ecosystem. When done effectively, decentralized systems promote value creation and equitable sharing rather than extractive practices.

Two-sided marketplaces like Airbnb, Uber, and eBay illustrate the centralization problem vividly. These platforms thrive on network effects, where their value grows as more users (buyers and sellers, drivers and riders, hosts and guests) participate. However, they are centralized entities that extract value by imposing fees and commissions. Decentralization offers a compelling alternative by reducing dependence on extractive intermediaries, allowing for fairer value distribution and governance by participants themselves.

Centralization, while advantageous for managing economies of scale, often leads to power consolidation, which increases the risk of corruption and value extraction. However, decentralization introduces its own set of challenges, including scalability, user adoption, and governance complexities. These systems require robust coordination mechanisms to avoid inefficiencies and ensure equitable value flows. Yet, recent advancements in blockchain technologies, smart contracts, and decentralized governance models demonstrate that these challenges can be mitigated. When implemented effectively, decentralization not only addresses power consolidation but also fosters innovation and resilience by aligning incentives across a distributed network.

Crypto-Economic Experiments​

At Matrix AI, we are actively exploring crypto-economic experiments using protocols like AAVE, Rocketpool, and Polykey as part of our broader investigation into decentralized systems.

Polykey, a decentralized secrets manager could evolve into a robust decentralized trust infrastructure. This infrastructure would incentivize not only the creation and maintenance of trust claims but also their verification and reputation management. By leveraging open reputation systems, Polykey could enable cross-border credit rails for world-wide peer-to-peer undercollateralized micro-loans, fostering trust and enabling financial inclusion at a global scale. Additionally, Polykey aligns with models like Gitcoin for public goods funding, where decentralized reputation systems ensure transparent allocation of resources. This vision extends to Polykey's Decentralized Trust Network, designed to coordinate trust and claims across a global, decentralized network, addressing enterprise needs while expanding into public authority applications with scalable and transparent solutions.

Our Zeta House project exemplifies how Matrix AI applies decentralized principles in the real world. In the future, Zeta House aims to expand into broader decentralized governance models. By enabling residents to hold stakes in their living spaces, Zeta House fosters community ownership, enhances liquidity, and ensures transparent governance. This approach bridges theoretical decentralized models with tangible outcomes, offering insights into the challenges and opportunities of implementing decentralized systems in everyday life. Future milestones include expanding its operational framework and integrating blockchain-based governance tools to further align resident incentives with shared goals.

Welcome to the era where AI isn't just augmenting our lives; it's beginning to commoditize our very thoughts, creations, and even our conversations. In this world, the bots don't just talk, they create, consume, and regurgitate content with such efficiency that the lines between original and synthetic blur. And, as this happens, something profound shifts in the digital landscape. Trust becomes the new currency.

We've all witnessed it. The explosion of AI-driven content generators, from GPT-4 and Claude to specialized coding bots like Aider, is transforming how content is created. These tools can whip up an article, solve complex coding problems, and even generate entire software modules, faster and cheaper than ever. This isn't just a revolution in productivity; it's a fundamental shift towards the commoditization of content itself.

For example, a colleague in clean energy consulting recently swapped original research for ChatGPT, using it as the foundation of all his presentations. The efficiency gains are undeniable, but so is the loss of originality. It's no longer about uncovering new insights; it's about remixing existing ones to suit the needs of the moment. And if we, as engineers, are honest with ourselves, how many of us have shifted from Stack Overflow to ChatGPT for problem-solving? The reliance on AI-generated content is becoming ubiquitous. It's just more specific and more convenient.

With everyone relying on AI to generate and consume content, a new question arises: what's the incentive to create human generated content if it's just going to be absorbed into the AI ecosystem? Bots consuming bots, an idea once relegated to dystopian fiction, is quickly becoming a reality. Content is generated, spliced, repurposed, and consumed, often without credit, copyright, or even a hint of authorship. It's content stripped down to its purest form—knowledge, ready for consumption, free from the constraints of origin.

This leads us to the "Dead Internet Theory" a concept where much of the Internet's content could soon be generated by AI, with humans playing a minimal role. More than a decade ago, I joked about creating an AI called Singularity to flood social networks with AI-generated noise until they collapsed. This is our reality now.

It is a world where LLM Optimisation (LLMO) replaces Search Engine Optimisation (SEO). Where website rankings are entirely irrelevant, because people aren't going to visit the original source when all consumption and generation of content will be mediated by AI.

The common retort to this, is that the majority of AI content will be garbage, and AI garbage/noise will overwhelm the good stuff/the signal. However this is only a problem when there's no discriminator of signal versus noise. Every content distribution system is a form of discrimination, that separates the noise from the signal. And the evidence shows that even before the advent of LLMs, the majority of content was garbage anyway. The same algorithms that bubbles up the good content will continue to work even in the world of LLM generated content. The biggest losers here are those that rely on content alone to make money. The question that must be asked is: where is the moat?

In a world where content is increasingly commoditized, the value of trust and authenticity skyrockets. The next wave of technological evolution isn't just about better AI; it's about creating systems that ensure the authenticity and provenance of the content we consume. At Matrix AI, we envision tools like Polykey becoming the decentralized trust infrastructure that democratizes access to provenance tooling for everyone. Forget blue checkmarks; we’re talking about a system where the very fabric of content is interwoven with verifiable trust.

We're moving towards an era where trust isn't just an add-on but a necessity. Imagine a future where every piece of content, every line of code, and every article is backed by an indelible record of its origin and modifications. That's where we're headed, and it's going to redefine how we interact with the digital world.

As we stand on the brink of this new era, it's essential to ask ourselves: How will we adapt? How will we maintain the balance between innovation and authenticity? The AI-driven commoditization of content is here to stay, but so is the growing demand for trust and provenance. We believe that the technological tools we're developing today, like Polykey, will play a pivotal role in shaping this future.

The bots are taking over, but it’s up to us to ensure that trust remains a fundamental part of the equation.

Apart from our professional endeavours, we're also committed to building a friendly and inviting cybersecurity community here in the Sydney CBD region. We do regular meetups and talks, where we aim to:

• Share ideas, best practices, and case studies
• Present the latest risks and developments in cybersecurity
• Practice cybersecurity problems and solutions
• Support each other via networking opportunities and broadening our business understanding of related topics

Diverse Community​

As a diverse community of cybersecurity professionals specialising in different fields, we intend our meetups to be an intersection of our combined knowledge. We encourage anyone to nominate topics of their fields of interest to discuss with the rest of us, whether it be technical or business focused, no matter how niche or mainstream. In all, we strive to create an accessible forum to learn, share, and connect with like-minded individuals.

Overview​

As a brief overview on what we've covered so far:

• In our initial meetup we presented Future Risks, Zero Trust, and the Optus Leak. Through it, we discussed the recent Optus Hack as a case study for the reasons as to the modern pivot towards Zero Trust architectures in the cybersecurity industry.
• Following this, we discussed the workings of Zero Trust Architecture and Electric Vehicle Cyberrisks, exploring the workings and possibilities of digital trust through a foundational understanding of game theory.
• To expand on our previous meetup, we discussed causes and risks of Secret Sprawl, and the implementation of Principle of Least Privilege secrets management workflows that mitigate this phenomenon.

We invite you to join us at future events by signing up to our Cybersecurity & Digital Trust Meetup Group.

Over the last several weeks, Matrix AI has been training the senior and principal software engineers at NetScout (NASDAQ: NTCT) on machine learning with a focus on deep neural networks.

It is a high-intensity course focused on bringing senior software engineers up to speed with the latest developments in deep machine learning. This will allow them to make practical use of machine learning techniques to solve big and small business data problems.

Roger's course was a great way to get up to speed on machine learning. The mathematical concepts introduced really made the subject matter stick. Now, we are capable of dissecting ML-based research papers, understanding what the innovations are, and applying them to problems we want to solve. -- David Turnbull - Principal Software Engineer

If your company requires leveling up your existing senior software engineers to be capable of understanding and making use of deep learning technology to solve real-world business problems, see our course page for more information (link).

Matrix AI is a company that applies Artificial Intelligence to DevOps and Cloud Infrastructure through the Matrix Operating System. We provide consulting services in the domain of machine-learning. This mean we help companies develop end-to-end machine learning infrastructure that encompasses data analysis, distributed big-data processing pipelines, bespoke information-extraction tools, deep neural network architectures focused on regression and classification, and GPU compute orchestration. Our specialty is in computer vision which involves image classification, object detection, segmentation and geographic information systems.

Machine learning for image classification and object detection sounds straightforward, until you're neck-deep in debugging misclassifications or training stalls. In this post, we'll unpack some real-world problems we've faced while working on the image classification of solar inverters for asset compliance auditing and our strategies to tackle them.

Convolutional Neural Networks (CNNs) are an impressive tool, but they aren’t magic. They’ve got quirks that can lead to subtle bugs or costly inefficiencies if not accounted for during training. Here's a closer look at some of these issues and how we dealt with them.

The Rotation Invariance Problem​

Rotating an image of a solar inverter slightly can throw a classifier off its game. CNNs don’t natively handle rotations well, as they learn patterns specific to the orientation seen during training. Our first solution was the obvious one: augmentation. But adding all possible rotations was computationally expensive and introduced diminishing returns.

Instead, we built an augmentation pipeline that rotated images to a canonical orientation before passing them to the classifier. This way, the classifier could focus on what it does best, detecting features, without being thrown off by rotations.

The Overlapping Class Problem​

Are two visually identical inverters always the same model? Not when government compliance regulations step in. This issue arises when classes overlap visually to the point that even a human would struggle to distinguish them. Our classifier often struggled with such ambiguity, leading to misclassifications that were technically correct in appearance but incorrect by regulatory standards.

To address this, we used a dual strategy of oversampling underrepresented classes and undersampling overrepresented ones. For oversampling, we augmented underrepresented classes with transformations such as rotations, scaling, and brightness adjustments. This artificially increased the diversity of these classes in the training dataset, improving the model's ability to recognize rare variants.

For undersampling overrepresented classes, we turned to a combination of the Goldberg image similarity metric and the FastPair algorithm. By applying these tools, we could identify and eliminate redundant images within the same class, effectively reducing the dominance of overrepresented classes in the dataset. The Goldberg metric quantifies similarity between images, while FastPair efficiently computes the dynamic closest pair problem, identifying which images to remove without excessive computational overhead.

This balanced the class distribution and forced the classifier to learn finer details that distinguish visually similar but semantically different classes. The result was a more equitable training set and improved performance across all classes, especially those prone to overlap.

The Unary Classification Problem​

Not every image fed into our pipeline contains a solar inverter, some might not even contain anything useful. Early on, we manually labeled our training set to include only relevant images. But let’s face it, humans make mistakes, and this process doesn’t scale.

Our solution was to implement a unary classifier, trained specifically to detect whether an image contained a solar inverter at all. We based this on the work of Chalapathy et al. in their paper "Learning Deep Features for One-Class Classification", which uses a loss function designed around compactness and descriptiveness.

These two principles, compactness and descriptiveness, are key to the unary classification challenge. Compactness ensures that embeddings of "in-class" samples, like solar inverters, are tightly grouped in the feature space, while descriptiveness ensures these embeddings remain distinct from all "out-of-class" samples, representing the vast and unpredictable variety of everything else.

This approach aligns conceptually with loss functions like center loss or triplet loss. Center loss minimizes intra-class variance by pulling features closer to their class center, and triplet loss separates embeddings based on anchor, positive, and negative sample relationships. Similarly, compactness penalizes embeddings that are too spread out, while descriptiveness works to maximize separation between the target class and the infinite set of "non-inverter" samples.

Unary classification is fundamentally different from binary classification. Binary classifiers have the advantage of defined, balanced categories, allowing clear decision boundaries to be learned. In unary classification, we deal with an inherently asymmetrical scenario: one well-defined target class against an undefined "everything else." This imbalance makes traditional binary loss functions unsuitable, as they can't account for the boundless diversity of irrelevant data in the background class.

Training such a model requires overcoming additional challenges. Noise in the dataset, such as mislabeled examples or edge cases, can confuse the classifier. Additionally, the dataset needs to be representative of the class's full variation to ensure robustness. To address this, we augmented our training dataset with diverse samples of solar inverters captured under varying conditions, ensuring the model learned the distinguishing features that generalized well.

By filtering irrelevant data early in the pipeline, the unary classifier acts as a gatekeeper, passing only relevant images downstream. This saves computational resources and enhances the efficiency of later tasks, such as fine-grained classification and object detection. It’s an essential component of building robust and scalable machine learning pipelines for real-world applications.

File Systems Aren't Just a Storage Problem​

Training large models on massive datasets is already a headache without slow file systems or flaky VPN connections piling on. Early on, we wasted hours restarting training jobs because of these bottlenecks. Enter Mothra: our content-addressed filesystem designed for high-speed access and stability.

Mothra hashes all inputs using multihash and stores them on S3, with a local disk cache to handle frequent reads. This improved stability and reduced training time significantly. By cutting out our reliance on NFS or EFS and their VPN dependencies, we streamlined our workflow and slashed costs.

Why It Matters​

These solutions aren't just clever hacks; they're investments in reliability. When you're building a system for something as critical as asset compliance auditing, every misclassification can lead to compliance violations or wasted man-hours. And let's not forget: AI systems that fail in high-stakes scenarios don’t just disappoint, they erode trust in the technology itself.

We'll continue exploring ways to improve model robustness and training efficiency, including diving deeper into augmentation strategies, transfer learning, and domain adaptation in future efforts.

For the full presentation, watch the YOW! Data 2019 talk:

Solar technologies have long been a pillar of sustainable energy, accounting for almost 10% of Australia's total electrical energy production in 2019. One of the incentives driving this adoption is the Small-scale Energy Certificate (STC) program offered by the Australian government, which offers discounts for the purchase of small-scale solar energy equipment such as solar panels and inverters.

A compliance industry has emerged from the introduction of these incentives in order to verify the eligibility of residential solar installations for STCs. This is because only a certain set of solar equipment are approved for STCs. When an application for STCs is incorrect either due to fraud or accident, the compliance industry is fined by the government. Therefore, this process is critical and involves reviewing a large amount of photographic evidence, which is time-consuming and prone to errors. But what if this didn't have to be the case?

Matrix AI has been consulting a client in this compliance industry to replace the manual process of checking photographic evidence of solar inverters with a computer-vision AI system.

Currently this process involves collecting photographic evidence along with the claimed solar inverter model through a mobile application, which is then queued up into a backend order processing system that is checked by human workers.

We designed a solution using a convolutional deep neural network to automatically classify the photographs of inverters.

Convolutional Neural Networks (CNNs) are a type of deep learning neural networks used for image classification that extract visual features from input images. These features are then weighted and used to classify the most likely solar inverter model.

In our case, the inverter classifier model is trained on a large dataset of 400,000 solar inverter images containing over 3000 solar inverter models collected over the last 10 years. By comparing the inferred model of the solar inverter to the claimed model submitted by the user, we are able to verify whether the submitted application is potentially fraudulent or contains a mistake.

As we started training the inverter classifier model, we encountered two problems:

• There were many solar inverter models were visually indistinguishable from each other.
• Photographs taken by claimants did not have to be of inverters, they could be photographs of anything.

We solved the first problem by creating a labelling system that could coalesce solar inverter models together, and the second problem by stacking a unary classifier that separates all non-solar inverter images first. We will talk about the unary classifier in a subsequent blog post.

Our computer vision AI system was presented to the client through a custom classification interface, the photographic evidence is shown on the left, and the inferred model is shown on the right. The potential models inferred from a given image are ranked from highest to lowest probability, and allows the user to reorder the ranking in case of false-positives, as well as coalesce models in case of visual similarity. These corrections will be taken into account for a subsequent retraining in order to improve the accuracy of the AI system.

With the deployment of the inverter analysis system, this is just one step towards full automation of the compliance workflow process. Further data integration is required for full automation as this system still requires a human in the loop.

Further adoption of AI technologies in the compliance industry can reduce the costs of adopting renewable energy, especially as government incentive programs in renewable energy require complex asset auditing and compliance.

For the full presentation, watch the YOW! Data 2019 talk:

Previously we had talked about service-centric networking. Since then, we have now implemented a prototype experiment implementing the ideas from Serval that demonstrates ICMP ping migration involving the Linux iptables and Network Namespaces.

As a reminder of what we're dealing with, today's cloud infrastructure is increasingly viewing a group of machines as a "service" rather than just hosts connected to the internet.

Abstracting machines into services allows easier horizontal scaling as we can load balance between instances of those services. Doing this sort of abstraction requires a service name that is not associated to a particular machine. This name can be a "virtual" IP address. However, just because you can remap a name to point to a different concrete address, doesn't mean the endpoints on the 2 sides of the communication can continue the conversation transparently. Many communication protocols are not stateless. Therefore, remapping a virtual service name may involve complex state tracking and synchronisation.

Our experiment involves a network structure like below:

+----------------------------------------------------------------+
|                                                                |
| +-----+     +-----+     +-----+   +-----+                      |
| |a.com|     |b.com|     |a.com|   |c.com|                      |
| +--+--+     +--+--+     +--+--+   +--+--+                      |
|    |           |           |         |                         |
|    |           |           |         |                         |
|    +-----------+-----+-----+---------+         Host            |
|                      |                                         |
|                      |                                         |
|            +---------+---------+                               |
|            |orchestrator bridge|                               |
|            +-------------------+                               |
|                                                                |
+----------------------------------------------------------------+

Each domain listed above is just a human-readable name associated with an ICMP ping service running in their own network namespace. They are all connected to orchestrator via virtual ethernet. The orchestrator currently runs as a network bridge. The entire experiment is only within a Linux host.

We enter one of the network namespaces and initiate a ping operation to another service using an abstract service name ( represented as an IP address). Then inside the orchestrator, we stopped the receiving service, and started another service while preserving the same abstract service name. The servicing sending ICMP ping continued to work without any interruptions.

This experiment does not address zero-downtime migrations, it was only an exercise in using Linux OS tooling for preserving abstract service names between different services.

The experiment also raised issues such as having a NAT engine which is able to load balance as well (which iptables doesn't do) as well as further questions on whether DNS resolution is a suitable mechanism for flow establishment, naming of services and migration of communication state.

The experiment source can be found at: https://github.com/MatrixAI/Relay/tree/master/experiments/stateless_container_ping_migrations

Working with type systems and communications can be very difficult. In Haskell, messages between threads are usually conveyed through channels that only accept a single type, and as a result are not very versatile. In networked systems, a stream of bytes is sent that hopefully conforms to one of many protocols including TCP, UDP, and HTTP, and hopefully both parties agree on what is sent and when. Session Types are a way to bridge the gap between these systems, as they can capture complicated protocols, and be properly type checked.

There are many ways that Session Types are notated and implemented. Scribble's notation looks more like an imperative programming language, where others are more functional, using algebraic data types and typeclasses. The notation in this post is inspired by a paper on Gradual Session Types and a paper on Multiparty Session Types.

The most basic units of Session Types are for sending and receiving data; !t represents sending a value of type t, and ?t represents receiving a value of type t. These can be concatenated by a semicolon, so a service that receives a string and responds with its length could be represented by the session type ?String; !Nat, and any client using it would be represented by !String; ?Nat.

To represent more complicated protocols we add the possiblity of branching, though it is useful to include in the notation which party is responsible for choosing the branch. For the party that is deciding on the branch we allocate the session type +{a:x,…,c:z}, where a and c are labels for the Session Types x and z. The party that is not responsible for choosing would have the corresponding session type &{a:x,…,c:z}. With this, we can encode a server that offers the client of receiving a guest page or signing in as a user with the following Session Type:

&{
  guest: !HTML,
  user: ?Username; ?Password; +{
    success: !HTML,
    fail: !Error
  }
}

The last feature of this version of session types is one that allows sessions to loop. For this we use the mu fixed point operator, which is notated as μt.a(t), where a(t) is some session type that contains t, and can be interpreted as the recursive definition t = a(t). If we wanted a user to have unlimited attempts at logging on to a system, our server could have the following Session Type:

μt. (
  ?Username;
  ?Password;
  +{
    success: !HTML,
    fail: t
  }
)

With all of these features, we can attempt to encode a HTTP 1.1 client, which can send as many requests as it wants before receiving responses:

µt. (
  !HTTPRequest;
  +{
    sendMore: t,
    wait: end
  };
  ?HTTPResponse;
  +{
    sendMore: t,
    wait: end
  }
)

Session Types have a notion of duality, where two protocols are compatible when they are duals. We define ?t to be dual to !t, &{a:x,…,c:z} to be dual to +{a:u,…,c:w} when all the choices x and u, z and w etc. are duals, and a series of concatenated session types x;…;z is dual to u;…;w when each pair of session types u and x, w and z etc. are duals. The dual of a fixpoint μt.a(t) is μs.b(s), where assuming t and s are duals implies that a(t) and b(s) are duals.

However, duality on its own does not define compatibility, since our earlier server that offered services to either a guest or a user would be compatible with a client that always chooses to be a guest, with the session type +{ guest: ?HTML }. To formalise this we define what it means for two Session Types a and b to be subtypes, notated a <: b, where the key property we want from subtypes is that if we have a <: b and c <: d and a and c are compatible, then the more general Session Types b and d should be compatible.

With this in mind, &{a:x,…,b:y} is a subtype of &{a:x,…b:y,…,c:z}, since offering extra options does not break compatibility, +{a:x,…b:y,…,c:z} is a subtype of +{a:x,…,b:y}, since choosing a subset of the available options does not break compatibility. Thus offering fewer choices is a subtype of offering more, and choosing between more choices is a subtype of choosing fewer.

To handle fixpoints, if assuming t <: s implies a(t) <: b(s), then μt.a(t) <: μs.b(s). Using this we can come up with a recursive definition of subtypes. Finally, we can define two Session Types to be compatible if one is a subtype of the other's dual.

An alternative syntax for the above operators is to replace ! with send, ? with recv, + with choose, & with offer, ; with significant whitespace, and μ with rec. Therefore the previous HTTP example would be like:

rec t (
  send HTTPRequest
  choose {
    sendMore: t,
    wait: end
  }
  recv HTTPResponse
  choose {
    sendMore: t,
    wait: end
  }
)

Most uses of Session Types are in allowing more complex communication within a strongly typed language, however at Matrix AI we are hoping to apply them to distributed systems. We will use Session Types to describe network protocols, allowing us to check which parts of a distributed system can communicate, to type check the construction of network combinators, and to analyse the behaviour of networked systems.

In 2016, Docker has officially updated their image specification from V1 to V2, adopting a more sophisticated scheme that is inline with OCI Container Image Specification.

There are only a few minor differences between Docker's image spcification V2 and OCI image specification (See Compatibility Matrix). Here we will discuss some major changes from V1 and V2, and why Docker has moved towards these changes.

Docker images contain the underlying changes in the root filesystem, and the execution parameters of a container. When we write a Dockerfile, the FROM clause bring in a base image from an external registry; each line of operation adds a new layer or attribute to the base image. Eventually docker build will gives us an image that we can deploy, share or run as a container.

It is important to distinguish that an image is different from a container. When a container is ran, the image (which is a stack of tarballs with manifest JSON files) is processed and transformed into the underlying filesystem, mounts, environment variables and so on. OCI Container Image Specification and OCI Container Runtime Specification give a good modern understanding on what an image is.

Content Addressability

In Docker Image Specification V1.0, each image layer has a randomly generated 256-bit id that uniquely references the layer. The space of the id is sufficient to ensure the uniqueness of all layers, but it does not guarantee that the image you pull is always the image you expect. Imagine that you have an image pychat:1.0 that uses python:3 as its base image. You uploaded your image to an image registry, and over the years someone comes along and swaps out some files in python:3. All in a sudden your code breaks and you might have to look into a deep chain of dependencies to figure out why your container doesn't work anymore. This is why having an unique id is not enough, we want a way to efficiently address the content of a container.

Content addressing is achieved by using a collision resistent hash function. In Docker Image Specification V2, each image layer is referenced with a unique content address computed from the canonical representation of the layer changeset. We can look at it like this:

"""
Note that this is an over-simplified version of how
image_ids are generated.
"""
layer_ids = []
for layer_changeset in layers:
    layer_id = hash(repr(layer_changeset))
    layer_ids.add(layer_id)
image_manifest = layer_ids
image_id = hash(image_manifest)

Due to the nature of cryptographic hash functions, the hashes can be used to distinguish whether two layers contain exactly the same content. We can use this property to ensure that the image we download is the same image we used before, i.e. tell Docker to fetch an image with the content address of a previously used and tested image. This will ensure that the image we get hadn't been changed since, as any changes to the image will result in a entirely different content address.

A manifest is a JSON file that contains all necessary configuration for an image. In Docker image spec V2, a manifest contains the content addresses of all layers in an image. Since it contains the hash of all layers, we can simply download this small JSON file and take the hash of it to verify that this image has all the layers we wanted. This saves us a lot of time from downloading and checking the content of the entire image.

An example V2 image manifest:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  ...
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 32654,
      "digest": "sha256:e692418e4cbaf90ca...b51fab815ad7fc331f"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 16724,
      "digest": "sha256:3c3a4604a545cdc12...f4a9c1905b15da2eb4"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 73109,
      "digest": "sha256:ec4b8955958665577...7f12184802ad867736"
    }
  ]
}

Example Dockerfile utilizing the image described in the above manifest:

FROM python@sha256:b22de77...f118cb
...
CMD ["python3", "pychat.py"]

This way pychat:1.0 is guaranteed to pull the same base image everytime it is ran, and the container will remain deterministic until our selected hash algorithm is found to be broken, which hopefully is not in the near future.

One thing to look out for when pulling images by content address:​

There are some inconsistencies between the content of the image manifest in the system and the content of the manifest that exists in the registry, i.e. The image ID displayed in docker image list --no-trunc is different to the address you use to fetch an image (relevant blog post and Github issue).

Image Specific Properties

Another important change is the location of image specific properties. In V1, each layer contains a JSON file that specifies what the image should do up to this layer. The file contains information such as entrypoint, env, cmd, memory and so on. However, these attributes should not be layer specific. Each layer essentially represents a change in the image's filesystem at a point in time, associating such component with runtime properties such as environment variables, entrypoints and CPU usage is completely unnecessary. Runtime properties should be related to an image as a whole, not a single layer in the filesystem.

We can take a look at Docker V1's Image format by doing docker save <image name> | tar -x in an empty directory. docker save and docker load are the two docker commands that still uses the V1 standard for legacy reasons.

$ docker pull ubuntu@sha256:d2518289e66fd3892c2dae5003218117abeeed2edbb470cba544aef480fb6b3a
$ docker save ubuntu@sha256:d2518289e66fd3892c2dae5003218117abeeed2edbb470cba544aef480fb6b3a | tar -x
$ tree
.
├── 08ca6384a97957eac5a5a69cdc799434739655c88e69efb23d2bb963110dbf48
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── 0fa211e5edebeb29d3e29cc2c8c87e9a6a8306901816c19b7f6fb6a7392c3cef
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── 452a96d81c30a1e426bc250428263ac9ca3f47c9bf086f876d11cb39cf57aeec.json
├── 614c02cb92ee20d3cd51770f07d67503f87a75602ddf032a0a6163527fcf97e0
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── cc8487ed6373e8b38c60ff8fc5bdfdd9576aa49226a6e4dcac522f61f5f19d31
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── daf8616e33b20539309a114814ba9864367630ad8da63d4e96bea40dd22841ba
│   ├── json
│   ├── layer.tar
│   └── VERSION
└── manifest.json

We can see that the image has 5 underlying layers (represented by sub-directories) and a top level layer (represented by the root directories). The manifest.json tells us the ordering of the layers and which of them is the top level directory. We can also see that there is a file json in each sub directory, this is the configuration of each layer in V1 format as described earlier. Read the json files and pay special attention to the container_config attribute set, these attributes should be assoicated to the image as a whole, however they exist at each layer instead. Having a hierarchy of inheritance on these runtime attributes layer by layer often leads to unnecessary complexity.

In V2, the layer JSON is discarded and instead it was decided that the layer changeset itself is enough to represent a layer in the image. Layer hierarchies are specified in the layers attribute in the root level manifest, which references the layers by its content addresses. The attributes of container_config is now stored in another file inline with the OCI Runtime Specification. The new specification makes sure that the runtime configurations are associated with the image as a whole, and inheritance happens at a image level rather than a container level. Note that the content address of this runtime configuration is also included in the V2 manifest.

Multi-Architecture Images​

V2 also introduces a new configuration component called a manifest list, or an image index in OCI's terminology. Manifest list is essentially a list of platform specific image manifests which contains similar contents, for example, an Ubuntu:16.04 container for a macOS host and an Ubuntu:16.04 container for a linux host. This allows multi-architecture containers to be coupled together and treated as a whole, which is useful for packaging a set of similar images and distributing them to an image registry.

Conclusion​

Here's a summary of the differences mentioned in this article:

• Image Spec V1
  • Randomly generated image ID
  • Image specific properties are defined at layer level
  • No multi-architecture support
• Image Spec V2
  • Image IDs are content addresses
  • Image specific properties defined at image level
  • Multi-architecture support

There are other advantages, we'll examine in later posts.

The Matrix AI team has been developing Polykey, a distributed peer-to-peer secret sharing system. It is intended to manage secrets, passwords, API keys for both humans and machines.

Many secret management systems have been designed either only for humans, or only for machines. We think this is unnecessary and intend Polykey to work in both cases. However, for this article, we'll focus solely on what Polykey provides to the Matrix infrastructure.

Matrix Automatons may require secrets in order to communicate to an external API. There are several challenges to managing infrastructural secrets:

• How to automatically deploy software which relies on external API keys?
• How to manage key updates while there are online applications using and depending on those keys?
• How to ensure the security of the keys at-rest or during transit?
• How to revoke access keys?
• How to only pass keys that are strictly relevant for the application? (Principle of Least Privilege)
• How to backup keys?
• Polykey was created to solve these problems and more. It integrates Git, public key cryptography and Keybase. It is designed to operate in a distributed manner similar to Git repositories (and using Git gives us version control for free). Here we will illustrate the abstract semantics of Polykey and in later articles we will address how we are implementing it.

A Polykey keynode is a single file. Keynodes can be stored anywhere. In this way, it is similar to pass. A keynode stores a set of secrets. Each secret is independently version controlled (this is where Git comes in). Each secret or subset of secrets can be shared to another keynode. You can create multiple keynodes and make some keynodes only store subsets of other keynodes. Sharing can be either push-based or pull-based (just like Git). For keynode A to push or pull secrets to or from keynode B, node A must have the capability to do so. This is done via public key cryptography.

In this way, a network of keynodes can represent a delivery mechanism for updating secrets. The computation required for dealing with secrets such as crytographic calculations or network transmission is divorced from where the secrets are stored. Furthermore, read-only computations only require read-only access.

Because keynodes are just files, they are completely self-hosted and self-contained, and you can be sure that your secrets are encrypted at-rest and during transit. They are also constructed using standard Unix formats: tar archives, git repositories, GnuPG encryption, Unix hardlinks, and just Plain Text. You don't need to launch a service or bind to some network. Secrets can just be acquired using a command line program, or directly accessed via a standard libraries that understand Unix file formats. This makes the Polykey keynode format very flexible and amenable to automation.

The exact design of Polykey is still under flux, but you can follow along here: [https://github.com/MatrixAI/Polykey]

We have been developing several libraries to support the development:

• js-resource-counter: Sequentially Allocatable and Deallocatable Resource Counter
• js-permaproxy: Proxy pattern for proxying an object mediated through a container.
• js-virtualfs: Virtual posix-like filesystem that runs completely in-memory.
• js-reference-pointer - Wrap a primitive value in a reference pointer, and pass by reference!
• js-object-tagger: Takes a POJO and tags specified keys pointing to objects with unique numbers.
• js-array-fixed: Arrays with Fixed Preallocated Length
• js-tree-order-index: (not yet working) - Indexing Trees for Efficient Queries and Updates
• js-virtualgit (not yet working) - In-memory Git

We expect Polykey to be usable on every platform (Linux, Windows, Mac, Android, iOS, Browsers). Even platforms that don't use JavaScript can just use standard Unix tools!

TCP/IP networking relies on IP addresses mapped to a machine to facilitate routing through the Border Gateway Protocol ( BGP). This mapping is usually done through the DNS, which maps human-readable names ("cats.com") to IP addresses. Service centric networks are an alternative/extension to DNS, where the host maintains their own table of mappings from service names to tuples of IP address and port number, but with additional capabilities to control the routing of data. This additional flexibility is usually referenced in the literature as separating the control plane and data plane. The key advantage provided by this is additional flexibility in dynamically modifying the service table compared to DNS. Our discussion of service centric networking is based on the Serval paper here.

In service centric networks, the control plane is the mechanism used to configure routes in the network, and also to configure the path that a payload may take through the network. Once the route for a payload has been established through the control plane, the payload is said to move through the data plane to the target based on the established configuration. The reason for this separation is that control and configuration of the network is itself distributed, and requires communication channels of their own.

One way this control/data plane split is achieved is to use a host maintained service table as an intermediary between the control plane and data plane: routing decisions in the data plane are made using rules that are listed in the service table, but the value of these routing rules is managed independently by a local daemon running on each node in the network.

Something that we can do with this control/data plane split is called 'late binding'. In DNS any peer that is looked up must already have a bound IP address in the DNS table. This is called early binding, because the IP address of the peer is explicitly written in the DNS table. With the service table, when a connect is called, the IP address of the peer can actually be discovered dynamically, by routing the initial packet of the connection through the network, to find a peer that supports the service. After the peer is found, the peer can open a connection directly to the host, and the connection is now established. This approach, for example, allows us to implement a load balanced network without a load balancer.

Payloads moving in the data plane have an additional layer of encapsulation between the addressing headers used to facilitate network forwarding, and the headers use to identify the protocol that is being communicated. This additional information consists of a source and destination 'flow' identifier, the transport protocol that is being used, a series of sequence and acknowledgement numbers, and a service identifier for communication. We can distribute a payload over multiple flows to and from the peer as necessary.

Our use case for service centric networks is to allow for finer control and addressing in a microservice infrastructure (more so than in DNS), and being able to change the location of such services on the fly. We would also like to make use of features implicit in the transport protocol that is chosen for a flow, that allow us to migrate live peers to a new location, and reattaching any open flows in a way that is transparent to the end user.

The fundamental isolation technology supporting containers on Linux are Linux namespaces. Namespaces provide isolation of global resources in a way that is transparent to the processes within the namespace. There are currently 7 different namespaces that are supported: Cgroups, IPC, Network, Mount, PID, User, UTS. Today we will be looking at the network namespace.

The network namespace can be thought of as a copy of the network stack. It provides isolation in network interfaces, routes, and firewall rules. In this article we will be using the tools from iproute2 to demonstrate.

First let's create a new network namespace to play with:

# Create a new net namespace ns0
$ ip netns add ns0

# Show all net namespaces
$ ip netns show
ns0

Here we have created a new network namespace called ns0. Let's go into the namespace and see what it looks like.

# Execute `ip link` in the ns0 namespace
$ ip netns exec ns0 ip link show
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

As shown from this example, our newly created namespace ns0 only has one loopback interface. If you are familiar with network interfaces (those "labels" you see in the well known ifconfig command), you can see that ns0 does not inherit any of the interfaces such as eth0, wlan0 from your main namespace.

This is what we mean by isolation of network device: whenever we create a new network namespace, the network resources in that namespace is separated from your main namespace. Processes that runs in this namespace (e.g. ip link) does not have access to the interfaces in other namespaces and vice versa.

Linux namespaces allow processes within one computer to think that they actually exists on different computers. In a network context, you can create a virtual network consisting of different processes, rather than per computer.

Let's get into the fun stuff and create a network of processes! There are a few ways to allow communication between containers, the most common way is to use a device called veth pair (virtual ethernet pair). In simple terms, veth pair is always created in pairs, and whatever information we passed into one end will come out from another. Let's start with a simple example to demonstrate veth pair.

# Let's create another namespace ns1
$ ip netns add ns1

$ ip netns show
ns1
ns0

Now we create a veth pair in the main namespace:

# Create a veth pair vth0 and vth1
$ ip link add vth0 type veth peer name vth1

And then we move the two end points into namespaces ns0 and ns1. Remember to check that our move operation was successful using the ip link command.

# Move vth0 into ns0 and vth1 into ns1
$ ip link set vth0 netns ns0

$ ip link set vth1 netns ns1

# Verify that vth0 is now in namespace ns0
$ ip netns exec ns0 ip link show vth0
6: vth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 9e:e4:e9:12:ad:42 brd ff:ff:ff:ff:ff:ff

# Verify that vth1 is now in namespace ns1
$ ip netns exec ns1 ip link show vth1
5: vth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether e2:da:f7:04:9e:e3 brd ff:ff:ff:ff:ff:ff

We also want to give those devices addresses and subnet.

# assign a static address to specified device
$ ip netns exec ns0 ip addr add "10.0.0.1/24" dev vth0

$ ip netns exec ns1 ip addr add "10.0.0.2/24" dev vth1

Let's bring the devices up and see if it works.

$ ip netns exec ns0 ip link set vth0 up

$ ip netns exec ns1 ip link set vth1 up

$ ip netns exec ns0 ping -c 2 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=0.047 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.052 ms

--- 10.0.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.047/0.049/0.052/0.007 ms

$ ip netns exec ns1 ping -c 2 10.0.0.2
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.055 ms

--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.043/0.049/0.055/0.006 ms

We have created a working virtual network between two network namespace. Veth pairs are easy to implement, but what if you want to add in more namespaces? it will eventually become a complete graph with a lots of links all over the place. This is very inefficient and hard to maintain. To solve this problem, we can use a bridge.

A bridge works like a virtual switch, it has the ability to connect multiple ethernet segments together in a protocol independent way. In our case, we are trying to connect multiple virtual ethernet segments together. Let's try putting the veth pairs we created earlier into a bridge.

# Create a bridge device in ns0
$ ip netns exec ns0 ip link add br0 type bridge

# Assign an IP to the bridge device
$ ip netns exec ns0 ip addr add "10.0.0.1/24" dev br0

# Brings the bridge up
$ ip netns exec ns0 ip link set br0 up

# Plug the veth end-point into the bridge
$ ip netns exec ns0 ip link set vth0 master br0

Notice here that we have set the bridge's address the same as vth0's address. This is because vth0's address does not matter anymore as it is now connected to br0, and br0 essentially serves as the entry of the network for ns0 now.

We should now revoke the address that was given to vth0 so the traffic can be routed properly.

# Revoke vth0's address
$ ip netns exec ns0 ip addr del "10.0.0.1/24" dev vth0

Let's test that it all works:

# Pinging 10.0.0.2 from ns0
$ ip netns exec ns0 ping -c 2 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=0.054 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.063 ms

--- 10.0.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.054/0.058/0.063/0.008 ms

# Pinging 10.0.0.1 from ns1
$ ip netns exec ns1 ping -c 2 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.052 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.066 ms

--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.052/0.059/0.066/0.007 ms

Now if we want to add a namespace into the network, we can simply do the following:

# Create a new namespace ns3 with a veth pair vth2 and vth3
$ ip netns add ns2

$ ip netns exec ns2 ip link add vth2 type veth peer name vth3

$ ip netns exec ns2 ip addr add "10.0.0.3/24" dev vth3

$ ip netns exec ns2 ip link set vth3 up

# Move vth2 into ns0
$ ip netns exec ns2 ip link set vth2 netns ns0

$ ip netns exec ns0 ip link set vth2 up

# Plug vth2 into the bridge
$ ip netns exec ns0 ip link set vth2 master br0

Then you can test the connectivity using ping. ns0, ns1 and ns2 should be able to ping each other successfully.

There are other ways to enable communication between network namespaces, such as Vlan, Macvlan, IPvlan, tuntap, etc. These will be left for a later topic.

At Matrix AI our research into language constructs for formally describing distributed systems has led us to a concept called Linear Temporal Logic. It is a logic system that allows us to express logical statements about time. This article will provide a brief introduction to this logic system.

Linear Temporal Logic takes the basic ideas of Propositional and Predicate logic and introduces concepts about time.

Predicate logic is widely used for automatically checking and proving theorems, verifying software correctness and security, and is the foundation of the programming language ProLog.

In Linear Temporal Logic (LTL), we can express statements about things that are true. Our expression will have both English word variants and also operator variants. The simplest statement is simply A. This means that A is true right now. We can also write that A will be true from now on and to the future, and that can be expressed as Forever A or □A. If A will be true eventually, we can write Eventually A or ◇A. Notice that Not Forever Not A is the same as Eventually A or ¬□¬A≡◇A. This is because if A won't always be false then A will eventually be true. If we model time as discrete moments, we can say that A will be true in the next moment, and write it as Next A or ◯A.

There are more sophisticated operators too, such as A Until B or A U B, which means A is true until B is true, and Eventually B. This can be written recursively as A U B ≡ B∨(A∧◇B∧◯(A U B)). There is also a weaker version of Until, denoted A W B, where B need not occur. We can write A W B ≡ □A ∨ (A U B).

There is a strong parallel between Propositional Logic and Functional Programming. Let's use Haskell as an example. If the formulas A and B are types then A∧B can be written the pair (A,B), A∨B can be written as Either A B, and if A implies B, we use the function type A -> B. Now function application looks like Modus Ponens, the distributivity laws look like alternative representations of the same types, and formal proofs look like Haskell programs.

As for the modal operators of linear temporal logic, we have to look at Functional Reactive Programming. Forever A is an unending stream of values of type A, Eventually A is a container that will eventually yield a value of type A, and A Until B is a stream of type A terminated with a value of type B. Using these types, IO operations can have types like Forever (Int, Int) for mouse position, Eventually Bytestring for receiving network data, and Char `Until` Nothing for standard input.

Evan Czaplicki's Paper goes over the many attempts to implement Functional Reactive Programming. However, for distributed systems, a complete functional reactive language is unnecessary. Instead, we can learn from Arrowised FRP, and model processes as Signal Functions. This model may help us describe the distributed services that Matrix is orchestrating.

Nix, NixOS and NixPkgs allows us to create project-specific development environments with project-specific dependencies (this usually means things like a C project, or a Python project... etc).

The way this is done is different for every language community within the Nix ecosystem. The most well developed patterns would be the C/C++, Haskell and Python community, other language communities tend to be smaller and has less documentation. This article serves as an introduction to using Nix for developing projects in different languages that we have worked with. This means we will only focus on shell.nix.

Remember that Nix is a turing complete purely functional language, and so it has real functions. Defining a project environment within Nix is done via functions. The most common function utilised for this purpose is stdenv.mkDerivation. However, various language communities within Nix has wrapped this function in higher order functions to provide further automation to deal with the environment configuration that different language toolchains expect.

First decide whether your project is an "application" or a "library". If it's an application, you need a default.nix and a shell.nix. But if it's a library you only need a shell.nix. The default.nix that is created may is generally not the same as the default.nix that gets committed into the mainline nixpkgs repository. Your project's default.nix should be a self-contained expression that can be used to build and install your project. Whereas the default.nix committed into the mainline nixpkgs needs to be compatible with the rest of the nixpkgs package set in order to merge into a "harmonious" package set. In practice this means the default.nix in your project will use a pinned nixpkgs package set, whereas the default.nix in nixpkgs will expect a pkgs parameter to be passed in.

The default.nix will be utilised when you run nix-build. The shell.nix will be utilised when you run nix-shell.

The shell.nix defines a development environment, so it is usually simpler than the default.nix. The default.nix needs to also define installation locations.

When encountering dependencies not available under nixpkgs, you have 2 choices. Either write a nix expression for that package, or utilise a language-specific package manager. In our examples below, we will utilise language-specific package managers within our shell.nix in order to create environments that work even under non-Nix environments. However, for dependencies that rely on system libraries, we'll generally install them via Nix, rather than doing compilation without or nix-shell. To make this work elegantly, we want to make sure we are not specifying dependencies twice, and some language-specific package managers are able to recognise dependencies already installed by Nix.

Nix 2.0 has the ability to verify hashes when using the builtins.fetchTarball function with the sha256 property. To know what hash you want to use. Run nix-prefetch-url --unpack https://package-you-want-to-use. In our examples, we utilise this parameter. However, to make the expressions work under 1.x, remove the sha256 parameter.

Often language specific dependencies do not appear in the online package search engine https://nixos.org/nixos/packages.html. To actually find available packages, you can either look at the nixpkgs source code, or use nix-repl with :l <nixpkgs> and then accessing the package set relevant to your language. The same package set may be aliased to multiple names. Some names are more general than other names. While the names may seem ambiguous, within a content-addressed package set, what these names map to is deterministic. In subsequent nixpkgs versions, the names may point to new package sets.

If you are making use of pinned nixpkgs following our previous tutorial: https://matrix.ai/2017/03/13/intro-to-nix-channels-and-reproducible-nixos-environment/ You can acquire the hash of your current pinning via: git -C /nix/nixpkgs rev-parse HEAD.

The package set for C in nixpkgs is itself!

Here is an example shell.nix for a C project:

{
  pkgs ? import (fetchTarball {
    url = https://github.com/NixOS/nixpkgs-channels/archive/084445b8f38ff8196f4b3f16d0ad0e79aa88dcbc.tar.gz;
    sha256 = "0jqxx3csxbs32ijn8w6cbd9c3l9vvsjz57rqsyp44dgg37xxx00i";
  }) {}
}:
  with pkgs;
  stdenv.mkDerivation {
    name = "c-project";
    buildInputs = [ autoreconfHook ];
  }

Out of all the language environments, C has the best support. The stdenv.mkDerivation already brings in the standard C toolchain.

In this particular example, we've used autoreconfHook as a buildInput. You only need this if your project uses Autotools. If you use CMake, you would instead bring in cmake.

JavaScript

The package set for JavaScript in nixpkgs is: nodePackages.

Here is an example shell.nix for a JavaScript project:

{
  pkgs ? import (fetchTarball {
    url = https://github.com/NixOS/nixpkgs-channels/archive/00e56fbbee06088bf3bf82169032f5f5778588b7.tar.gz;
    sha256 = "15pl5p3f14rw477qg316gjp9mqpvrr6501hqv3f50fzlcxn9d1b4";
  }) {}
}:
  with pkgs;
  stdenv.mkDerivation {
    name = "javascript-project";
    buildInputs = [ nodejs-8_x flow ];
  }

Note how we are utilising nodejs-8_x. This is just an alias. What the alias means depends on the content addressed package set. We are also bringing in flow which is a JavaScript type checker.

Within this project we can use npm freely just like normal. So that's where we will put all our dependencies.

For most projects, this is enough. However, some projects require building C++ extensions. To do this we need to find the toolchain required to build the extension. The 3 common choices are node-gyp, node-gyp-build and node-pre-gyp. Look at the package.json of the package you're trying to install to find out which one is being used. Once you know this, you can add the relevant toolchain to the buildInputs.

# you don't actually need all 3 gyp toolchains, this is just an example
buildInputs = [
  nodejs-8_x
  flow
  nodePackages_8_x.node-gyp
  nodePackages_8_x.node-gyp-build
  nodePackages_8_x.node-pre-gyp
];

In some cases, you may also need python. It all depends on what kind of package you are installing.

The resulting project works in a Nix environment, and also works outside of a Nix environment.

Python

The package set for Python in nixpkgs is: python2Packages, python3Packages.

Here is an example shell.nix for a Python project:

{
  pkgs ? import (fetchTarball {
    url = https://github.com/NixOS/nixpkgs-channels/archive/630dbfe672164eeaf4bc822226cbb3ad7c1b0805.tar.gz;
    sha256 = "0xvj0z928mzcs56hindxw608a6jm1fvsi2ap5r7m4a0plnrcwx90";
  }) {}
}:
  with pkgs;
  python35Packages.buildPythonApplication {
    name = "python-project";
    buildInputs =  (with python35Packages; [
      numpy
      gdal
      scikitimage
      tensorflowWithCuda
      (matplotlib.override { enableQt = true; })
    ]);
    shellHook = ''
      echo 'Entering Python Project Environment'
      set -v

      # extra packages can be installed here
      unset SOURCE_DATE_EPOCH
      export PIP_PREFIX="$(pwd)/pip_packages"
      python_path=(
        "$PIP_PREFIX/lib/python3.5/site-packages"
        "$PYTHONPATH"
      )
      # use double single quotes to escape bash quoting
      IFS=: eval 'python_path="''${python_path[*]}"'
      export PYTHONPATH="$python_path"
      export MPLBACKEND='Qt4Agg'

      set +v
    '';
  }

This one is more sophisticated because it shows how to get Tensorflow with CUDA, and matplotlib rendering with Qt4.

Because pip by default installs into a global directory, we have to set PIP_PREFIX to a Project local directory. This directory should then be ignored by .gitignore.

When using pip, it actually is aware of dependencies installed via Nix. So in this case, if you later try to install a package that depends on numpy, pip will not reinstall numpy. Unless of course that dependency has a constraint that is not met by the numpy that you installed. The point is you can still write a normal setup.py and requirements.txt, and the project is still usable by non-Nix developers.

Haskell

The package set for Haskell in nixpkgs is: haskellPackages.

Here is an example shell.nix for a Haskell project:

{
  pkgs ? import (fetchTarball {
    url = https://github.com/NixOS/nixpkgs-channels/archive/8b1cf100cd8badad6e1b6d4650b904b88aa870db.tar.gz;
    sha256 = "1p0xxyz30bd2bg0wrfviqgsskz00w647h0l2vi33w90i42k8r3li";
  }) {}
}:
  with pkgs;
  haskell.lib.buildStackProject {
    name = "haskell-project";
    buildInputs = [];
    shellHook = ''
      echo 'Entering Haskell Project Environment'
      set -v

      alias stack="\stack --nix"

      set +v
    '';
  }

This example is really simple. Because here we are relying on stack to bring in all the Haskell dependencies similar to how we setup a shell.nix for JavaScript.

The usage of haskell.lib.buildStackProject wraps stdenv.mkDerivation and adds all these features: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/generic-stack-builder.nix

When using --nix flag on stack, it just makes sure that stack will utilise the GHC supplied by the haskell.lib.buildStackProject, instead of acquiring its own GHC, which would be a waste of time. All other stack commands work normally.

Becareful with the usage of the alias here. If you have scripts (like a Makefile) inside that call stack, they may not have --nix applied. If so something will go wrong. It would be better to have some sort of environment variable instead, but I'm not aware of it.

An alternative to using the alias is to set:

nix:
  enable: true

In your Project's stack.yaml or in ~/.stack/config.yaml. If you are expecting that your projects will be used by people not using Nix, then it makes sense not to put it in the Project's stack.yaml. However instead you can then put it in ~/.stack/config.yaml.

When using stack if a dependency doesn't build, that could be because it's missing a C library. The most common is probably zlib. If so, just put it into your buildInputs.

Emscripten

The package set for Emscripten in nixpkgs is: emscriptenPackages.

{
  pkgs ? import (fetchTarball {
    url = https://github.com/NixOS/nixpkgs-channels/archive/084445b8f38ff8196f4b3f16d0ad0e79aa88dcbc.tar.gz;
    sha256 = "0jqxx3csxbs32ijn8w6cbd9c3l9vvsjz57rqsyp44dgg37xxx00i";
  }) {}
}:
  with pkgs;
  stdenv.mkDerivation {
    name = "emscripten-project";
    buildInputs = [
      emscripten
      cmakeCurses
      pkgconfig
      python2
      nodejs
      flow
    ];
    shellHook = ''
      EMSCRIPTEN_ROOT_PATH='${emscripten}/share/emscripten'
      EMSCRIPTEN='${emscripten}/share/emscripten'
    '';
  }

At this moment we are not aware of any function which wraps stdenv.mkDerivation that deals with emscripten requirements. Preferably this can either be stored in a package hook when you bring in emscripten or it can be placed into a special function. This means at this point if you bring in an emscriptenPackages, it isn't automatically registered. You have to find the relevant environment variable to configure in your shellHook.

PHP

The package set for PHP in nixpkgs is: phpPackages.

Here is an example shell.nix for a PHP project:

{
  pkgs ? import (fetchTarball {
    url = https://github.com/NixOS/nixpkgs-channels/archive/084445b8f38ff8196f4b3f16d0ad0e79aa88dcbc.tar.gz;
    sha256 = "0jqxx3csxbs32ijn8w6cbd9c3l9vvsjz57rqsyp44dgg37xxx00i";
  }) {}
}:
  with pkgs;
  stdenv.mkDerivation {
    name = "php-project";
    buildInputs = [
      php71
    ] ++ (with php71Packages; [
      composer
    ]);
  }

Here we rely on composer to bring in all the PHP dependencies. So it's just like JavaScript.

Database Services

Application projects often might rely on a database service. You can use shell.nix shellHook to help set this up, so that each project can have its own database running within the shell. To make this work without adding in container/network namespaces, we rely on unix domain sockets. This allows each database process to only communicate with a project local unix domain socket.

Here is an example using MySQL and the Flyway migrations system:

shellHook = ''
  echo 'Entering MySQL Environment'
  . ./.env
  set -v

  alias mysql="\mysql --socket='$(pwd)/.mysql/mysql.sock' "$DB_DATABASE""
  alias mysqladmin="\mysqladmin --socket='$(pwd)/.mysql/mysql.sock'"
  alias mysqld="\mysqld \
    --datadir="$(pwd)/.mysql" \
    --socket="$(pwd)/.mysql/mysql.sock" \
    --bind-address="$DB_HOST" \
    --port="$DB_PORT""

  alias flyway="\flyway \
    -user="$DB_USERNAME" \
    -password="$DB_PASSWORD" \
    -url="jdbc:mysql://$DB_HOST:$DB_PORT/$DB_DATABASE" \
    -locations=filesystem:$(pwd)/migrations";

  set +v
'';

Notice the .env file which stores all the environment variables. So you can use shell.nix as a dotenv replacement.

Remember to set up initialise your database before you start using it!

rm -rf .mysql && mkdir .mysql mysqld --datadir="$(pwd)/.mysql" --initialize-insecure

launch your database without networking!

mysqld --skip-networking &

Again beware of using aliases, it is actually better to use environment variables when possible as they will carry over to any scripts you run.

Here's a bonus using PostgreSQL and PostGIS:

{
  pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/49a0ecc49b491c1f1a6f329d3d4fc9cf559b18aa.tar.gz) {}
}:
  with pkgs;
  let
    pg = (pg:
      buildEnv {
        name = "postgresql-and-plugins-${(builtins.parseDrvName pg.name).version}";
        paths = [ pg pg.lib (postgis.override { postgresql = pg; }) ];
        buildInputs = [ makeWrapper ];
        postBuild = ''
          mkdir -p $out/bin
          rm $out/bin/{pg_config,postgres,pg_ctl}
          cp --target-directory=$out/bin ${pg}/bin/{postgres,pg_config,pg_ctl}
          wrapProgram $out/bin/postgres --set NIX_PGLIBDIR $out/lib
        '';
      }
    );
  in
    stdenv.mkDerivation {
      name = "project-with-postgres-postgis";
      buildInputs = [ (pg postgresql) ];
    }

This introduction assumes you have played with NixOS a bit, you know about content addressability and why it is important, and how Git repositories represent a distributed content addressed storage system.

Git and Github is used as the source control for all of NixOS and NixPkgs. Both NixOS and NixPkgs source code is located here: https://github.com/NixOS/nixpkgs. This means every package that is available via Nix is defined in that repository. This includes OS services and general software. Nix, the language interpreter and the package manager tool is however located here: https://github.com/NixOS/nix.

Channels is a rolling distribution endpoint that provides the latest builds of NixOS and Nixpkgs, and the Nix expression set corresponding to those builds. Because NixOS is just a particular build of NixPkgs that is geared towards running a full OS, we'll be using NixOS channels and not NixPkgs channels. The channels have names of the form:

• nixos-YY.MM
• nixos-YY.MM-small
• nixos-unstable
• nixos-unstable-small
• nixpkgs-unstable

We will ignore NixPkgs channels (nixpkgs-*) and only focus on NixOS channels (nixos-*) because the NixPkgs channels are intended to be used by non-NixOS users like other Linux distributions or Macintosh users. The exact artifacts that the channel points to will have their names detailed in this way: label-YY.MM.N.H-*. The label is is name of the artifact such as nixos, nixos-graphical, nixos-minimal. The N is the iteration number of the major version YY.MM. The H is a 7 character truncated Git hash. The * is the reamining metadata for the build artifact.

Channels are built from Git release branches on the official NixPkgs source repository: https://github.com/NixOS/nixpkgs. The branches are notated as release-YY-MM. On every commit to these branches, Hydra (the Nix build system) clones the source code and builds them while running tests. If these tests succeed, the successful branch commit is then propagated to a channels repository located at: https://github.com/NixOS/nixpkgs-channels. Hydra will also "release" them to the official channel endpoint at: https://nixos.org/channels/ . Once a commit is available at https://nixos.org/channels, it is guaranteed that the packages specified inside the corresponding Nix expressions will be built, and cached in the binary cache.

Let's see if these things match up (here we use httpie, pup and jq):

http https://nixos.org/channels/ | pup 'table tr:nth-child(n+4) td a attr{href}'

http 'https://api.github.com/repos/NixOS/nixpkgs/branches?per_page=100' | jq '[.[] | select(.name | startswith("release"))]'

http 'https://api.github.com/repos/NixOS/nixpkgs-channels/branches?per_page=100'

The released channel hashes (https://nixos.org/channels/) exactly match the channels repository branch hashes (https://github.com/NixOS/nixpkgs-channels).

However because Hydra takes time to build and verify release branches, the release branches at https://github.com/NixOS/nixpkgs will be sometimes ahead of the released channel branches and released channels.

Some extra things to note:

• The small channel variants are updated before the entire package set is built. This means the small channel variants will be up to date with the latest security changes and bug fixes compared to the non-small variants. This does mean that sometimes Nix will force a build from source rather than downloading pre-built packages.
• The nixos-unstable channel corresponds to the master branch at https://github.com/NixOS/nixpkgs. The master branch will sometimes be ahead of the channel, as the channel relies on Hydra to finish evaluation.
• There is no channel that corresponds to the staging branch.
• The staging branch at https://github.com/NixOS/nixpkgs is even more unstable than master. The staging branch is intended for changes that will cause mass-rebuilds of packages, and is supposed to be merged back into master every couple weeks.

So which source should we source our channels from? It depends:

• If you are developing on NixOS, you'll often need to make use of the latest commits or pull requests, in that case, you have to use branches from https://github.com/NixOS/nixpkgs.
• If you're just using NixOS and want to have a stable experience, you should use the official released channels endpoint at https://nixos.org/channels/
• If you're power user, but you still want a stable experience, you should use the Git release branches at https://github.com/NixOS/nixpkgs-channels.

The great thing about Nix, is that you can mix-and-match. Your system can be sourced from release branches, while cherry-picking changes directly from official nixpkgs repository, and even picking up forks, random commits and making use of them inside a Nix profile or a nix-shell.

However if you need to download the built ISOs for a given release, these should be downloaded from https://nixos.org/channels/, because the repositories don't host these artifacts. Another source for built artifacts is Hydra itself, but we'll not go into this yet.

It is at this point where we must differentiate the channels at https://nixos.org/channels/ and the Git repository sources from which those channels are built from. The minimal definition of a "channel" is a directory with 2 files: nixexprs.tar.xz and binary-cache-url. The nixexprs.tar.xz is the tarball that contains a default.nix which is what is actually used to evaluate Nix expressions. The binary-cache-url is just a text pointer to another location containing the cache for the Nix expressions.

The main utility of using these official channels over the sources in the Git repositories is so that you can know when the binary cache is ready to supply all the packaeges in the channel's Nix expression. Nix provides a command called nix-channel which allows you manage system and profile channels. The system channel is also the root user's channel.

Ultimately you can think of using Nix channels for every day usage, but you should be using the Git repository hashes as sources for your package sets when you care about reproducibility, which basically whenever you're developing software. On installation of a fresh NixOS system, there will be one channel the system is already subscribed to.

Running sudo nix-channel --list will show something like: nixos https://nixos.org/channels/nixos-17.03. The nixos name is just an alias, it allows you refer to multiple channels during package installation. (This is not as flexible as directly using the Git sources, because you need a proper channel structure including the binary-cache-url, and we don't have a valid binary cache for every single source commit hash.)

Having multiple channel aliases, allows you perform installations like nix-env --install --attr nixos.packagename or nix-env --install --attr mycoolchannel.packagename. This also means anybody can publish a channel, they just need to provide a binary cache as well.

We had to use sudo because by default in order to see the channel of the root user/system. But every user on a NixOS system can manage their own set of channels, where the channel aliases can override the system channel aliases.

# make sure you don't have `.` or `-` in the channel name!!!
# this is a bug: https://github.com/NixOS/nix/issues/1457
nix-channel --add https://nixos.org/channels/nixos-17.03-small nixosSmall1703
nix-channel --update nixosSmall1703
nix-env -iA nixosSmall1703.xmlstarlet

Adding a channel by itself does not download the channel expressions, you need explicitly run the update command. The channels are stored in /nix/store but referenced from ~/.nix-defexpr (this exists for both normal users and the root user). If you want to learn more about this, watch the contents of ~/.nix-defexpr/channels/ and ~/.nix-defexpr/channels_root/ while running the above commands.

The nix-channel command is quite limited, it does not tell you what channels are out of date, so you would need to run the --update command often to get the latest released iterations of each channel. You can find the exact version of your channel by looking at cat ~/.nix-defexpr/channels/nixosSmall1703/svn-revision or ~/.nix-defexprs/channels_root/nixos/svn-revision. But this is not an official supported technique.

You can remove channels with nix-channel --remove nixosSmall1703, and you don't need to run nix-channel --update.

Because of how channels work, we prefer to side-step this functionality and instead pin our NixOS system a particular commit hash within a released channel. This is what makes a NixOS system reproducible. We'll try to make sure that the packages we want to install are all from a single commit hash. Note that this is sometimes not possible when a package at a particular iteration or version is only available at a different commit hash, and you don't want to move your entire system to that commit hash. However with Nix, it's very easy to have packages from multiple different commit hashes cohabiting on the same system, and when you need isolation, this can be provided via Nix profiles or nix-shell.

While we could just import the desired Nix expression inside our configuration.nix, certain tools like nix-env relies on the ambient channel setup to install packages. The right way to solve this would be to make nix-channel support specific commit hashes and local directories containing Nix expressions, rather than only channel directories. This would make pinning a NixOS system much easier, and would seamlessly work with the existing nix-env expectations. There is however a workaround involving the utilisation of the Git content-addressed system and NIX_PATH. This workaround was discovered here: http://anderspapitto.com/posts/2015-11-01-nixos-with-local-nixpkgs-checkout.html ( There is an error with this workaround that is discussed below.)

# switch to super user
sudo --login
# we're going to create /nix/nixpkgs as our pinned nix expressions
rm --recursive --force /nix/nixpkgs
git clone https://github.com/nixos/nixpkgs /nix/nixpkgs
cd /nix/nixpkgs
git remote add channels https://github.com/nixos/nixpkgs-channels
git fetch --all
git checkout -B channels-nixos-17.03 channels/nixos-17.03
# /nix/nixpkgs will now be set to branch channels-nixos-17.03 and tracking channels/nixos-17.03

Now go into your configuration.nix, and set nix.nixPath = [ "nixpkgs=/nix/nixpkgs" "nixos-config=/etc/nixos/configuration.nix" ];.

Since the current NIX_PATH hasn't updated yet, we need to force a rebuild with the the new source for Nix expressions: nixos-rebuild -I nixpkgs=/nix/nixpkgs switch.

When you want to update you can just perform a git pull on the directory. These Git commands will also be helpful when you want to checkout a specific commit hash, or when you want to switch to a different channel source.

# show remote info
git remote show channels
# update all remotes
git fetch --all
# see all the branches that is available
git branch --all --verbose --verbose
# see what the remote provides along with pull requests to that remote
git ls-remote channels
# if you have added changes to your local branch, you can rebase your changes on top of any updates
git rebase channels/nixos-17.03 channels-nixos-17.03
# if you want to reset to upstream channel and drop any local changes (resets the working tree as well)
git reset --hard channels/nixos-17.03

Now we can remove the system channel and any user-channels as they will no longer be used:

nix-channel --remove nixosSmall1703
sudo nix-channel --remove nixos

All nix commands except the nix-env command will work. The nix-env currently only uses ~/.nix-defexpr. This issue (https://github.com/NixOS/nix/issues/993) asked to make nix-env use NIX_PATH or <nixpkgs> as a fallback if there are no channels installed. However this was only implemented in the new Nix UI project, and is not available on the current nix-env (1.11.15). The only way to make nix-env use NIX_PATH is to alias it with the option --file '<nixpkgs>'. This is safe as you can override the --file option by specifying it again. Once you do this, attribute path installation doesn't require a channel name. For example: nix-env --file '<nixpkgs> --install --attr 'hello'.

While this makes sense for general purpose use, for unattended NixOS servers where it doesn't make sense to use nix-env, there is no need to do all of this, and you can just directly import a content-addressed Nix expression set inside your configuration.nix.

At this point we can see how to acquire packages from different sources (including other commit hashes) and have them all coexist. There are 2 ways to do this. Through the imperative nix-env command that mutates your profile state, and through Nix expressions that you can inline into your configuration.nix or your nix-shell configuration files.

nix-env --file '<nixpkgs>'
nix-env --file '/package.nix'
nix-env --file '/directory/containing/default.nix/'
nix-env --file '/tarball/containing/default.nix/tarball.tar'
nix-env --file 'https://github.com/NixOS/nixpkgs/archive/master.tar.gz'
nix-env --file 'https://github.com/NixOS/nixpkgs/archive/74f22ff827d7c91fe26a62c69a53556a62ecc70c.tar.gz'
nix-env --file 'https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz'

Because /nix/nixpkgs is owned by root, if you run Git operations via sudo, you'll end up with permissions that don't allow usage by other users (due to the default umask used by Git when creating new files). You'll need to run these commands to reset to proper permissions. We need to make sure that all directories are at least can be readable and executable by all users, and that all files are at least readable by all users.

they are just build support files

sudo chmod --recursive u=rwX,g=,g=rX,o=,o=rX /nix/nixpkgs

There may be a way to avoid these permission problems in the future.

As you can see we rely on Github to expose specific commit hashes as tarballs that our nix-env can consume. This makes it convenient to install packages from different branches, forks and even pull requests.

The same thing can be done inside a Nix expression such as our configuration.nix. The key primop is import. This command expects either a directory containing default.nix or a Nix file. If we intend to acquire an expression set remotely, we just need to use fetchTarball to expose the Nix expression. Note that nixpkgs based expressions is exported as a function, which means you can apply an empty attribute set to it before using it. The attribute set argument is intended to provide various flags into the Nixpkgs attribute set.

let
  pkgs-env = import <nixpkgs> {};
  pkgs-my = import ./package.nix;
  pkgs-dir = import ./packages/containing/default.nix/;
  pkgs-unstable = import (fetchTarball http://nixos.org/channels/nixos-unstable/nixexprs.tar.xz) {};
in
  {
    environment.systemPackages = [
      pkgs-env.packageAttrName
      pkgs-my.packageAttrName
      pkgs-dir.packageAttrName
      pkgs-unstable.packageAttrName
    ];
  }

Any path specified with angle brackets such as <nixpkgs> utilises the NIX_PATH environment variable to find files. We just set our NIX_PATH to nixpkgs=/nix/nixpkgs, which means <nixpkgs> just resolves to /nix/nixpkgs. However if we instead had <something/subthing>, this would try to look for something/subthing in each of the paths listed in the NIX_PATH. This is how it works:

NIX_PATH = nixpkgs=/nix/nixpkgs
<nixpkgs> = /nix/nixpkgs

NIX_PATH = nixpkgs=/nix/nixpkgs
<nixpkgs/lib> = /nix/nixpkgs/lib

NIX_PATH = nixpkgs=/nix/nixpkgs
<nonexistent> = Error

NIX_PATH = /somedir/containing/something/:nixpkgs=/nix/nixpkgs
<something> = /somedir/containing/something/something

You can find out ahead of time whether Nix will find an angular path by using: nix-instantiate --eval --expr '<something>'.

The same techniques can be further applied in your shell.nix when developing inside an isolated Nix shell, you should make sure your software environment is reproducible.

However if you are developing a package for the purpose of inclusion into official Nixpkgs, after you've done with development, if you are dependent on packages that is already available in one of the Nixpkgs iterations, you should rely on the ambient Nixpkgs provided through the pkgs attribute. This ensures that packages within a single Nixpkgs package set work in harmony with each other, and reduce the number of unnecessary duplicated dependencies. We have found that within the Nixpkgs source, there are times when multiple versions of a particular package is required by different downstream packages, this is when the particular iteration of Nixpkgs will often contain multiple versions of the same package. However, as soon as these conflicts are fixed, either by the upstream of the Nixpkgs maintainers, the older versions of the package will be deprecated and removed from future iterations of NixPkgs.

One possible downside is that you won't get automatic security updates with this approach. However this depends on your priorities. When in the act of developing software, reproducibility matters more. When in the act of maintaining/operating software, automatic security updates matter more. You can get the best of both worlds by building a system to monitor security issues relevant to your software dependency graph, and trigger updates then builds then tests before finally deploying.

UPDATE: We have found a way to avoid needing to apply permission changes after updating your /nix/nixpkgs. You do this by applying a temporary umask change. The reason is that the permission problems applies to new files being created by git.

we use a subshell to make umask change temporary

(umask 022 && sudo git -C /nix/nixpkgs checkout channels-nixos-17.09) (umask 022 && sudo git -C /nix/nixpkgs pull)

The above can be automated by changing sudo umask options: https://unix.stackexchange.com/a/278817/56970

Also, when you are using sudo while doing git operations requiring authentication. You may require sudo -E or sudo --preserve-env to preserve your SSH or GPG environment.

Configuration divergence is a result of change entropy in an environment that doesn't enforce referential integrity between loosely-coupled state or "state-at-a-distance". Simultaneously this allows the anti-pattern of action at a distance. This phenomenon is also called "Connascence" and I have also called this "meta-coupling".

This process gives us a system that becomes more and more opaque as time continues, and we lose referential transparency on the entire system. The result of course is a repeated cycle of entropic reduction projects (often called " refactoring" or "rewriting") where consultants are hired to clean up technical debt. Occasionally these projects will suffer from the second system effect. All of this resolves to increase the cost of changing/adapting software systems.

With this in mind, the significant difference between data and code is that code is usually considered to be "stateless", and data to be "stateful" , but this is not a difference in kind, but a difference in degree. In time, both data and code will be changed, and will exhibit all the properties of "stateful" systems. Therefore, this problem cannot be solved with tools that only enforce a consistent baseline. We either accept divergence and try to converge back to the fixed point, or we rebuild with tools that are correct by construction. As a software community, we'll get there eventually, but it's going to take time.

Here are examples of this problem:

• Magic Strings and Magic Numbers
• Laravel's Route Tags

Today, the most common way for developers to address a dependency is through the use of semantic versioning. This is where you create project A that depends on B at 1.2.3, where 1 is the major version, 2 is the minor version and 3 is the patch version.

Accordingly, the rules of following semantic versioning means:

1. Major versions change when you make incompatible API changes.
2. Minor versions change when you add functionality in a backwards-compatible manner.
3. Patch versions change when you make backwards-compatible bug fixes.

Theoretically you should be able to update patch and minor versions of dependencies without any kind of problem. In fact many package managers recommend that you should use the "next significant release" dependency address notation of ~1.2 which means >=1.2 <2.0.0 or ~1.2.3 meaning >=1.2.3 <1.3.0.

In practice this often can lead to subtle bugs, and occasional flaky builds. There's just nothing that guarantees that updating a patch version will not break a build, and sometimes software is so complex, the runtime behaviour of a minor version bump might cause problems in production or at scale. Over time we've learned that dealing with these nightmares is not worth it, and that fixed versions is the right way to go. If we want to update, we must verify the update works. This idea of fixing versions is done by many package managers, such as PHP's composer will create a composed.lock that will lock down all the versions as long as you don't run another composer update.

However, there's still one more problem, there's no guarantee that what is 1.2.3 today is what's 1.2.3 a month from now. This is because there's no direct relationship between a semantic version tag and the actual contents of the dependency. Instead, there's just a third party that you are trusting when you are asking them for the package that is 1.2.3. The only constraint that 1.2.3 is enforcing, is that the package must say it's 1.2.3. However the package could have changed in the meantime, while spoofing the package version.

If only there was a way to refer to some package or dependency, not by some arbitrary tag, but by an address that represented the contents of the package...

Well there is! Check out Content Addressable Storage. The idea is simple, we need a crytographic hash function to generate a digest of the package contents, store in into a location that is referencable by hashes.

While this is not a new idea, doing a short google search shows that there is only 3 package management systems today that can be considered a content addressable package system:

• Nix, Nixpkgs and NixOS (and the related Guix)
• ied
• Git

Having content addressable dependencies is the first of the necessary factors for reproducible builds. The second necessary factor is for mirrored dependency storage. You cannot rely on the third party to maintain availability of the dependency (they may go down for a myriad of reasons: downtime, ddos, network problems, lawsuits like the leftpad fiasco).

Nix and Git are the 2 systems that I'm most familiar with in the above list. It turns out both systems can satisfy both of the necessary factors, but not out of the box.

For Nix, while we get content addressable dependencies. The official nixpkgs does not do any dependency or source code mirroring (it does have a binary cache, but that's not what I'm talking about). This means from time to time, you can get build failures because the upstream source code has gone offline, in fact this happened to me a month ago with the dependencies from freedesktop.org. However you can work around this, by creating your own mirrors of the upstream dependencies, and creating nix expressions for them. It is however not currently a first class feature.

For Git, content address package management relies on the cutting edge submodule feature of Git. So cutting edge, that you need the latest Git 2.8 to get everything to work. Submodules are basically subproject Git repositories within a superproject Git repository. In fact you can actually clone any Git repository, and you can do work on that subproject and push from it to the upstream remote without affecting the superproject other than updating the commit hash that the superproject tracks. The key feature is that the superproject tracks the commit hash of the subprojects, hence all dependencies here are identified by their content address. So that when you later clone the superproject, you will get exactly the dependencies that were specified by the content address, and nothing else. Git currently uses a SHA1 hash as their commit hash.

Here we demonstrate a simple way of using Git submodules:

> cd /tmp
> mkdir --parents SuperProject
> cd SuperProject
> git init
Initialized empty Git repository in /tmp/SuperProject/.git/
> echo 'Hello World' > ./README.md
> git add --all
> git commit --message='Hello World'
[master (root-commit) a60c612] Hello World
 1 file changed, 1 insertion(+)
 create mode 100644 README.md
> mkdir --parents modules
> git ls-remote https://github.com/CMCDragonkai/bashpp.git
5e94d3c285f4424f3fa4c6ddf036f405688b1d51        HEAD
5e94d3c285f4424f3fa4c6ddf036f405688b1d51        refs/heads/master
5e94d3c285f4424f3fa4c6ddf036f405688b1d51        refs/tags/0.0.1
> git submodule add https://github.com/CMCDragonkai/bashpp.git modules/bashpp
Cloning into 'modules/bashpp'...
remote: Counting objects: 344, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 344 (delta 0), reused 0 (delta 0), pack-reused 338
Receiving objects: 100% (344/344), 66.20 KiB | 46.00 KiB/s, done.
Resolving deltas: 100% (170/170), done.
Checking connectivity... done.
> git diff --staged
diff --git a/.gitmodules b/.gitmodules
new file mode 100644
index 0000000..10f797f
--- /dev/null
+++ b/.gitmodules
@@ -0,0 +1,3 @@
+[submodule "modules/bashpp"]
+       path = modules/bashpp
+       url = https://github.com/CMCDragonkai/bashpp.git
diff --git a/modules/bashpp b/modules/bashpp
new file mode 160000
index 0000000..5e94d3c
--- /dev/null
+++ b/modules/bashpp
@@ -0,0 +1 @@
+Subproject commit 5e94d3c285f4424f3fa4c6ddf036f405688b1d51
> git add --all
> git commit --message='Added bashpp as a dependency.'
[master fc545ee] Added bashpp as a dependency.
 2 files changed, 4 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 modules/bashpp
> git submodule status modules/bashpp
 5e94d3c285f4424f3fa4c6ddf036f405688b1d51 modules/bashpp (0.0.1)
> git ls-tree master modules/bashpp
160000 commit 5e94d3c285f4424f3fa4c6ddf036f405688b1d51  modules/bashpp
> cd modules/bashpp
> git checkout cefe74b6d6006ee75897790ed2090b8bc7741ed9
Note: checking out 'cefe74b6d6006ee75897790ed2090b8bc7741ed9'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at cefe74b... Changed shebang to be more generic
> cd ../..
> git status --verbose --verbose
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   modules/bashpp (new commits)

--------------------------------------------------
Changes not staged for commit:
diff --git i/modules/bashpp w/modules/bashpp
index 5e94d3c..cefe74b 160000
--- i/modules/bashpp
+++ w/modules/bashpp
@@ -1 +1 @@
-Subproject commit 5e94d3c285f4424f3fa4c6ddf036f405688b1d51
+Subproject commit cefe74b6d6006ee75897790ed2090b8bc7741ed9
no changes added to commit (use "git add" and/or "git commit -a")
> git add --all
> git commit --message='Fixed bashpp dependency to cefe74b6d6006ee75897790ed2090b8bc7741ed9'
[master dacb700] Fixed bashpp dependency to cefe74b6d6006ee75897790ed2090b8bc7741ed9
 1 file changed, 1 insertion(+), 1 deletion(-)
> git submodule status modules/bashpp
 cefe74b6d6006ee75897790ed2090b8bc7741ed9 modules/bashpp (0.0.1~2)
> git submodule update --remote --merge modules/bashpp
Updating cefe74b..5e94d3c
Fast-forward
 bin/bashpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Submodule path 'modules/bashpp': merged in '5e94d3c285f4424f3fa4c6ddf036f405688b1d51'
> git status --verbose --verbose
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   modules/bashpp (new commits)

--------------------------------------------------
Changes not staged for commit:
diff --git i/modules/bashpp w/modules/bashpp
index cefe74b..5e94d3c 160000
--- i/modules/bashpp
+++ w/modules/bashpp
@@ -1 +1 @@
-Subproject commit cefe74b6d6006ee75897790ed2090b8bc7741ed9
+Subproject commit 5e94d3c285f4424f3fa4c6ddf036f405688b1d51
no changes added to commit (use "git add" and/or "git commit -a")

What we did above was to create SuperProject, added a submodule called bashpp which was put into modules/bashpp. We didn't specify any further constraints when it was added, therefore it was cloned into the default branch master. However, we could then go into the submodule, and checkout a specific commit hash (for content addressing), branch (for rolling updates) or tag (for semantic versioning), however for proper content addressable dependencies, you should prefer the commit hash, and use git ls-remote to acquire the commit hash for any particular tag or branch. Upon doing so, the superproject recognised this, and realised that the submodule is now tracking that specific version.

Subsequently, you can push this to Github, and clone it along with its submodules at exact their associated commit hash that you previously commited to the SuperProject. Here's a demo:

> git clone <repo>
> cd <repo-path>
> git submodule update --init --recursive --depth 1

The above should acquire the transitive closure of dependencies, and only at depth 1, that is not acquiring all dependencies. Currently however it the --depth 1 is a bit flaky. So it may be just safer to do git clone --recursive <repo> which will acquire the entire repository for all the submodules.

However, there is a feature that is currently missing, which is that the ability to acquire a clone/submodule of a Git repository at a specific commit and only at that commit, meaning a commit depth of 1. Currently as you see above, we have to acquire the entire development repository prior to fixing it at a particular commit hash. There's a discussion regarding this feature (or lack there of) here: http://stackoverflow.com/questions/26135216/why-isnt-there-a-git-clone-specific-commit-option . Ideally in the future, we can just do git submodule add --branch dadbf912508272c268c809f97955e182854f79be --depth 1 <repository> [path] in order to acquire a dependency at exactly that commit hash. It does preclude us from being able to develop within that submodule, but you should do that directly in a different clone.

Furthermore, the tooling for manipulating the hashes of the Git submodule called "gitlink" is still not fleshed out, at the moment we can only read the gitlink using git ls-tree or git submodule status, but not change the "gitlink" directly. We have to instead indirectly change the gitlink via updating the submodule repository HEAD reference ( using git checkout or git reset commands).

More information:

• http://stackoverflow.com/questions/27188899/shallow-clone-with-submodules-in-git-how-to-use-pointed-commits-and-not-latest
• http://stackoverflow.com/questions/1777854/git-submodules-specify-a-branch-tag
• https://matthew-brett.github.io/pydagogue/git_submodules.html
• http://stackoverflow.com/questions/2155887/git-submodule-head-reference-is-not-a-tree-error ( For dealing with the situation where the superproject refers to a subproject commit that no longer exists.)
• https://git-scm.com/docs/git-submodule

There's quite a few more features to submodules including recursive operations, where submodules contain further submodules. Therefore, as a feature it's still got quite a few rough edges, and the workflow isn't fully fleshed out.

In comparing the 2 workflows Nix and Git, if you have a NixOS environment to deploy into, you should definitely prefer to use Nix style. However, if you want your project distributable to people who aren't using NixOS, Git submodules is a viable alternative to get started using content addressable dependencies.

For both of them, it's important to consider the second necessary factor to reproducible builds. For Nix, make sure to mirror the source. For Git, you can just fork the project, prior to adding them as a dependency. However I'm looking forward to the integration of IPFS with Nixpkgs.

Now does this mean semantic versioning is out, and dadbf912508272c268c809f97955e182854f79be is in? No. Semantic versioning is still very useful as a metatag of software releases. It communicates intent of the developers, and that's very handy. But it's not something we should be relying on when building production software.

In this post, I haven't addressed the "harmoniousness" of a package set (the idea that packages in package set should work together out of the box with no conflicts). But this is solved Nix and Nixpkgs as well. And it also solves the problem with duplicate dependencies by flattening the hierarchy and hardlinking dependencies with the same hash. Nonetheless, it still requires hardwork from a community in making all the packages in a package set work without problems, so it's an ongoing effort.