This is a Patreon-supported essay. Drafts of all of these essays go out a week early to my $10 and up subscribers. Building secure applications is hard, and for organizations that have never done it before, it's often unclear where to even start. Worse, organizations that have some development experience often underestimate the work required to ship secure code. My goal with this essay is to make the landscape more legible and give NGOs and other organizations an idea of where to start. This is a four part essay; part two is here, part three is here, and part four is here.
I have a Patreon, here, where you can subscribe to support my security and systems-focused writing. You sign up for a fixed amount per essay (with an optional monthly cap), and you'll be notified every time I publish something new. At higher support levels, you'll get early access, a chance to get in-depth answers to your questions, and even for more general consulting time.
© 2021 Eleanor Saitta.
Building software securely is hard, even for organizations that know the people using their tools face real risks and that have the best of intentions. Figuring out what “doing well” looks like is even harder for organizations with limited technical experience, and even experienced development teams that haven't dealt with security-sensitive systems and processes in the past can run into trouble. This guide is targeted specifically at NGOs, but it should be useful for all sorts of organizations.
This is a high-level overview. This guide will not tell you how to do the activities it talks about, but rather will show you how the landscape fits together and give you enough of an overview of where the pitfalls, hard crags, and deep bogs are that you can begin to plan your journey. There is a long, long list of information that could have been included here but wasn't. I may add and revise this further after all four parts are out, but at some point all things must ship, and I also figure that even if I had infinite time, there's value to a short guide instead of a book, too.
This guide has four parts. In this, part one, we'll cover the software development lifecycle, organizational maturity, participatory design, security design, development documentation, requirements, and selecting a design team and security designer. In part two, we'll look at threat modeling for requirements and for architecture, architectural design, protocols and parsers, cryptography, selecting cryptographers to work with, selecting a development team, frameworks, standards, and third-party libraries. In part three, we'll go over per-module requirements, QA testing, automated testing and continuous integration, code and security reviews, and selecting and working with a security review team. The final part will cover field trials, shipping code, system updates, training, feedback, incident response, operational security for development teams, and long-term maintenance.
All software has to pass through a number of stages on the way to the group of people that will use it. These stages are called the software development lifecycle.
First, the development team works with folks who will be using the system to figure out what should be built. Most of the decisions that determine whether the software will result in better security outcomes for users happen here. This understanding is then turned into a set of requirements, some of which have to do with security.
At this point, threat modeling starts to make sure that the security requirements are complete and appropriate, given the goals of the system and the contexts it will be used in. We'll talk more about threat modeling in part two. Next, the high-level description of the system is turned into a description of its architecture, or how the functionality will actually be implemented. The next stage of threat modeling happens here, to make sure that the architecture specified can deliver on the identified security objectives.
Once the architecture is understood and the set of protocols, frameworks, and standards that are required for implementation have been determined, development begins. The threat model and security requirements, along with the rest of the architecture and design documentation determine the requirements for each module in the system and should be referred to by developers. As development on each module is completed, it is tested to ensure that it correctly implements all of the requirements and security objectives. Once the entire system is complete, it's tested as a whole, and further low-level testing is done looking for places where the specifications may have been incomplete or other code-level bugs may have been introduced.
Once the software has passed all the tests, users are trained on how to work with it and it's deployed to the field, generally in stages to detect any problems. New problems will almost inevitably be found after it's shipped, either in code built by the development team or in third-party software libraries they used. When this happens, a fix must be developed, people using the tool notified, and an updated version of the system sent out.
This is a simplified view of how the process works. In practice, most modern software teams use some form of agile development where parts of the software are finished and shipped before the entire system is done (the older model is sometimes called “waterfall”, because everything flows through once in big steps). In agile processes the same set of security practices still happens, but instead of being done once, all the security and development process artifacts become living documents that are continually updated and worked with as the process continues. Agile processes place a premium on solid low-level frameworks, peer review between developers, and careful choices in the early design process which are clearly communicated to the entire team. Done properly, agile processes can result in systems that are just as secure as traditional waterfall development. The shift from the heavy gatekeeper mode of interaction with a security team to frequent, small interactions can reduce organizational stress, help speed up knowledge transfer, and let mistakes be caught when less work has been done — all the reasons why teams adopt agile processes to start with.
No organization will go from zero to hitting every one of the practices we're discussing here in one go. Few large, well-funded software organizations in the commercial world hit them all either. That said, establishing policy and process around all of these issues, measuring your own progress, and attempting to improve is a critical part of delivering systems reliably and securely.
If you deliver a one-off project with no process or policy around it, it will almost invariably fail to produce the security benefit you want. It may even fail to provide any efficacy for users at all, and poorly structured projects often don't deliver anything at all. However, most of these processes require practice and culture change and that does not happen overnight. These processes are also designed to be checks and balances on the rest of the development process. This makes consistency and validation critical, both of which require maturity in execution. If you don't know where you are in terms of security maturity, chances are good that you're making mistakes.
Even if you're standing up an entirely new development team with no existing culture, there are limits to how many security practices you'll be able to build into your core process to start with. Attempting to build in too much up front (unless you have significant competence in other parallel development groups) is liable to cause over-planning and lead to slow processes and a poorly-gelled team. It's important to prioritize which practices you start with. I recommend security design, thorough threat modeling, a careful selection of the frameworks you build on, third-party security review, and an experienced team to work with around incident response as the minimum set of practices for delivering code to a high-risk environment. Each of these practices looks at security from a different perspective and complements the others. While there are more things that can and should be done with time, this is a minimal baseline.
If you aren't in a position to build a mature delivery organization internally, you need to work with one externally. We'll talk about this in part two under “Selecting a Development Team”.
Building tools that support better security outcomes for high-risk groups requires a deep understanding of what those groups are trying to accomplish. Too often, development teams build tools to support what they think people should be doing and not what the people actually need to do. The details of user community/team cultures need to be understood deeply and carried throughout the entire development process.
The best tool for this is participatory design, because no one knows more about what a group of people are trying to accomplish than those people themselves. This doesn't mean that users design the tools or that development teams just build what they ask for. While people know what they're trying to accomplish, it doesn't necessarily follow that they know the best way to support their own work. In participatory design, the designers work with people who may use the system to understand the tasks they want to accomplish in the world and their goals for those tasks, figure out potential solutions collectively, and then ensure that the users understand the solutions and agree that they're likely to improve outcomes.
Tools like personas (descriptions that are intended to be representative of real users) are useful, but are no substitute for engaging real users in the design process. Every abstraction loses information about the thing it represents, and it is hard for design teams to know which information will matter. While at some point in the process, choices must be made and abstractions introduced, the later this happens in the process the better. In general, the less like the user base the development team looks, the more user engagement is required to satisfactorily represent user needs. If the development team lives different lives and works in different ways than the people the system is for, they will need more ongoing contact to keep those lived experiences in the front of their minds while they work. In many cases, designing tools properly will require serious study.
In a more agile process, not all design will be completed at the beginning of the project. Instead, it will be an ongoing, iterative process, which has real advantages. The goal of initial design work in an agile process is to define a minimum viable product, which is to say the simplest thing that can support people in achieving some of their goals. To support agile processes in a participatory context, the development team must have long-term, on-going contact with the user community. If they do, they'll have the huge advantage of being able to see how people work with the real system, not just prototypes, while development is still happening.
For some teams, the participatory mode will encompass the entire design process. In other cases, it will only include the initial concept and feature definition, and more traditional user-centered design processes will be used for the graphical design of the system, task completion testing, etc. More participation is generally better, but as involving users can significantly increase the expense and complexity of the process, trade-offs are inevitable and reasonable.
All of this should standard practice for any system development initiative, but it's much more important for tools intended to improve security outcomes because the stakes for using the system being built are often much higher.
Building software that helps people accomplish their goals when they have adversaries means building software that has specific security properties selected to make a difference in that work. It also means building systems that enable effective work in general, including ensuring that people want to take initiative to use the system or interact with others via it. The process of selecting and defining the security properties a system needs to implement and the ceremonies, or user actions that must be performed to maintain them, is called security design. Security design doesn't stop at the security of the system, however. Well-done security design helps shape the way communities and teams work together to ensure that their activities are effective despite their adversaries.
Security design should, whenever possible, be done as part of a participatory design process. Understanding what properties will be useful in a given situation is often not easy for designers who don't understand the cultural context other people are working in. Indeed, because security properties deal with inherently exceptional events, even people from that context may need time, care, research, and opportunities for simulation or experimentation to figure out what they might need. When security design is done collaboratively, novel solutions often result, yielding tools that are far more useful than anything designed in isolation could be. Security design is as much designing, enabling, and supporting actions within a user team's culture as it is the system itself. It's impossible to do if you don't start by understanding and accepting the culture and structure of interactions the people you want to help already have.
The security design process requires a working understanding of what kinds of adversaries people are facing, what resources those adversaries have at their disposal, how those adversaries are likely to use their resources, and how the situation is likely to change over time. It also means understanding the resources people have at their disposal and the strategies they already use to avoid their adversaries. Much of this information may be confidential and sometimes extremely sensitive. Proper discretion in the security design process is critical.
Agile security design can be challenging. Retrofitting new security properties into an existing system often implies complete rewrites or redesigns. However, there are times when layering and abstraction can allow different systems to be swapped out later in the development process. If this is going to be attempted, a full understanding of the intended final security properties should still be considered a mandatory part of the initial minimum viable product design. These security properties will inform many design choices throughout the life of the system.
If a system is developed with security properties being added later, the development team has an ethical (and possibly legal) onus to ensure that no user uses the system for real production work until all security properties are delivered. It's recommended that all versions of systems being developed in this way are both clearly marked and partially functionally disabled during the testing period to prevent insecure versions being used in production by accident.
If the changes in the design process are less significant than adding and removing entire security properties, though, there can be benefits to the agile design approach. Group interactions are often complex and subtle, and even simple applications have a lot of designable surface that can be used to shape those interactions. In many cases, it may be necessary to watch or talk with people who are using an already-mostly-functional version of the system to seen the kinds of interactions it supports in more detail. From that position, one can rework the system to support a more diverse set of roles, to make certain common or outcome-critical actions easier, or to make interaction patterns between actors that lead to good security outcomes more legible.
Many development teams, especially teams newer to development or working on one-off projects, will wait until fairly late in the development process before they begin to write documentation. This is a critical mistake and a mode of working which is largely incompatible with the participatory design process and with developing secure software. While sensitivities in user interviews must be taken into account, raw interview materials, the insights drawn from them, and the design principles and feature ideas that result from the process all must be documented. Each time a decision is made, it should both be recorded and the rationale for the decision linked to the field observations or user comments that drove it. If this process is followed someone who is trying to understand later why a choice was made or what the intent was has the original sources close at hand.
It's important that features and decisions can be tracked through the entire development process and beyond. It may be the case that years later, a maintenance coder is attempting to fix a bug in the system and, without proper documentation, might accidentally break something critical to the system. It's obvious that this can happen at the technical level, so more mature projects maintain technical documentation. The equivalent issues at the design and requirements levels are less commonly understood.
It's also important to keep documentation as lightweight as possible to ensure it is navigable and kept up to date. This is a hard tradeoff that requires care and experience. Excepting security concerns, it's probably worth keeping more raw interview material around than less, even if one can't index all of it at first, because it can't be regenerated. Beyond this, various different trade-offs can be made to work as long as the trackability requirement for design and implementation decisions is maintained.
The intent of the design process is to develop the requirements for the system. In an agile process, this will often take the form of a set of user stories that will be implemented and added to piecemeal until the system reaches a somewhat final form. In other processes, there may be a single requirements document that describes the entire feature set of the system to be implemented. Requirements should be kept as simple as possible while still conveying the full intent of the system to be built. It is generally more useful to keep design research and user interview artifacts separate but discoverable. The more fully-specified the system requirements are, the more successful the threat modeling process will be. In particular, any security properties the system intends to implement and the user actions required to maintain them must be fully specified in the requirements documentation. If this is not done, the threat modeling process will not be able to correctly diagnose system security. Building good requirements documentation is an art to itself, however, and there's much more to consider. Requirements documentation drives the entire development process, so it's worth doing right.
If you're just building out your development team, it's important to carefully consider who you bring in for your design team.
First of all, you must have a design team.
Tools written entirely by developers are almost without exception unusable by people who are not also developers. Unless (and often even if) developers are your target market, your application will be useless if other people cannot use it, by definition. Yes, there are some amazing unicorns out there that can both design and develop, but chances are good that you are not one of them. Please work with a design team if you want your tools to have an impact on the world.
There are four basic roles within a design team. Depending on the size of the project, these may be different people, different teams, or a single person. That said, it will be tough to find a single designer qualified for all four roles.
The first role is a design ethnographer or participatory designer. They'll work directly with the communities that may use your tool, traveling if necessary, working through translators when required, and help drive the understanding of what people are trying to do. While the project lead must be involved as well, someone who understands how to work with a community, build trust, and gather this kind of qualitative data will be essential. Participatory designers should have experience working in contexts similar to the one the product is targeting.
The second role is a security designer. The security designer understands the available set of security properties and the problem space the participatory design process has uncovered. They work ensure the system is designed with a coherent and complete set of security properties that will function in the context as desired. They'll also work with the other design roles to make sure the system encourages participation and supports the work people are doing. The security designer is likely to be the most technical person on your design team. They should also lead the threat modeling effort and help coordinate other high-level security activities. Security design is a relatively new field, especially when considered as a separate discipline, and in many cases it may be necessary to cross-train someone into this position. While in theory going in either direction should work, in practice it will probably be easier for someone with existing design experience to learn about high-level security and to liaise closely with a security engineer than it will be for a security engineer to learn to design. Either way, though, doing this work well takes years of experience on both sides of that split.
The third role on the design team is the graphic designer and/or front-end developer. This is what many folks think of when they hear “designer”. They're responsible for the look and feel of the system and for it supporting the patterns of interaction that have been designed. For applications designed with security in mind, they're also responsible for making sure that the system correctly communicates the security properties it provides, the user actions required to implement them, and when failures have occurred. In large projects there may be a separate graphic designer or graphic design team from the folks who implement the front-end interface. On smaller teams, the graphic designer may also do front-end coding. Interface design for the web and for local applications is different, as is design for desktop and mobile. It's important that your front-end designer understand the environment they're designing for. Most interface designers have more web or mobile experience than desktop application experience these days.
The last role on the design team is the interface tester. It's not enough to just design an interface; it also must be tested to ensure it works as well for people as the team thinks it will. Design is a communication problem, and if an interface is clumsy or ambiguous, it will cause problems in the field. The interface tester is responsible for making sure the design delivers. Like design, interface testing for mobile, desktop, and mobile and desktop web are all quite different. Ensure that your tester can work in the environments you're building for.
Many teams may have a design lead responsible for integrating the different design functions. This role may also be taken on by the product manager for the entire system, depending on how the team is organized. In many teams, the design ethnographer, security designer, and interface tester may only be present for a small period in the overall system development process, in which case it's especially critical that the design lead can correctly represent their viewpoints. If you've got a small project where your design lead is also doing front-end development and graphic design, chances are you'll still need to bring in other folks for the participatory, security design, and interface testing portions. While all of these skills fall under the big umbrella of “design”, they're as different as QA and development are. If you have one person trying to wear too many hats, quality will suffer. Security design in particular has an outsized impact on the field efficacy of projects attempting to improve security outcomes. Many of the shortcomings we've seen over the past decades with security tools repeatedly proving near-useless in the field come from security design failures. Don't take shortcuts here and undermine all of the rest of your work.
When hiring a design team for a security-critical project, it's important that the team have experience with high-risk systems. Bringing a team that has no high-risk experience in to build a system that people will take real risks while using is a sure-fire way to get folks hurt. Not everyone need have the same level of experience, and in larger teams, having senior and junior staff working together is fine. In a small team, the most critical positions are the security designer and the design ethnorgrapher — they absolutely must understand the high-risk world. Most advertising and media agency design teams are categorically unsuited for this task, regardless of what they try to sell you. Insist on an experienced design team.