New legal limits on surveillance in the US
The United States has new legal limits on electronic surveillance, both in one specific way and — more important — in prevailing judicial theory. This falls far short of the protections we ultimately need, but it’s a welcome development even so.
The recent Supreme Court case Carpenter v. United States is a big deal. Let me start by saying:
- Most fundamentally, the Carpenter decision was based on and implicitly reaffirms the Katz test.* This is good.
- The Carpenter decision undermines the third-party doctrine.** This is great. Strict adherence to the third-party doctrine would eventually have given the government unlimited rights of Orwellian surveillance.
- The Carpenter decision suggests the Court has adopted an equilibrium-adjustment approach to Fourth Amendment jurisprudence.
- The “equilibrium” being maintained here is the balance between governmental rights to intrude on privacy and citizens’ rights not to be intruded on.
- e., equilibrium-adjustment is a commitment to maintaining approximately the same level of liberty (with respect to surveillance) we’ve had all along.
- I got the equilibrium-adjustment point from Eugene Volokh’s excellent overview of the Carpenter decision.
*The Katz test basically says that that an individual’s right to privacy is whatever society regards as a reasonable expectation of privacy at that time.
**The third-party doctrine basically says that any information of yours given voluntarily to a third party isn’t private. This includes transactional information such as purchases or telephone call detail records (CDRs)
Key specifics include: Read more
| Categories: GIS and geospatial, Surveillance and privacy | Leave a Comment |
Brittleness, Murphy’s Law, and single-impetus failures
In my initial post on brittleness I suggested that a typical process is:
- Build something brittle.
- Strengthen it over time.
In many engineering scenarios, a fuller description could be:
- Design something that works in the base cases.
- Anticipate edge cases and sources of error, and design for them too.
- Implement the design.
- Discover which edge cases and error sources you failed to consider.
- Improve your product to handle them too.
- Repeat as needed.
So it’s necesseary to understand what is or isn’t likely to go wrong. Unfortunately, that need isn’t always met. Read more
| Categories: Analytic technologies, Text | 3 Comments |
Brittleness and incremental improvement
Every system — computer or otherwise — needs to deal with possibilities of damage or error. If it does this well, it may be regarded as “robust”, “mature(d), “strengthened”, or simply “improved”.* Otherwise, it can reasonably be called “brittle”.
*It’s also common to use the word “harden(ed)”. But I think that’s a poor choice, as brittle things are often also hard.
0. As a general rule in IT:
- New technologies and products are brittle.
- They are strengthened incrementally over time.
There are many categories of IT strengthening. Two of the broadest are:
- Bug-fixing.
- Bottleneck Whack-A-Mole.
1. One of my more popular posts stated:
Developing a good DBMS requires 5-7 years and tens of millions of dollars.
The reasons I gave all spoke to brittleness/strengthening, most obviously in:
Those minor edge cases in which your Version 1 product works poorly aren’t minor after all.
Similar things are true for other kinds of “platform software” or distributed systems.
2. The UI brittleness/improvement story starts similarly: Read more
Technology implications of political trends
The tech industry has a broad range of political concerns. While I may complain that things have been a bit predictable in other respects, politics is having real and new(ish) technical consequences. In some cases, existing technology is clearly adequate to meet regulators’ and customers’ demands. Other needs look more like open research challenges.
1. Privacy regulations will be very different in different countries or regions. For starters:
- This is one case in which the European Union’s bureaucracy is working pretty well. It’s making rules for the whole region, and they aren’t totally crazy ones.
- Things are more chaotic in the English-speaking democracies.
- Authoritarian regimes are enacting anti-privacy rules.
All of these rules are subject to change based on:
- Genuine technological change.
- Changes in politicians’ or the public’s perceptions.
And so I believe: For any multinational organization that handles customer data, privacy/security requirements are likely to change constantly. Technology decisions need to reflect that reality.
2. Data sovereignty/geo-compliance is a big deal. In fact, this is one area where the EU and authoritarian countries such as Russia formally agree. Each wants its citizens’ data to be stored locally, so as to ensure adherence to local privacy rules.
For raw, granular data, that’s a straightforward — even if annoying — requirement to meet. But things get murkier for data that is aggregated or otherwise derived. Read more
| Categories: Derived data, Public policy | 2 Comments |
Some stuff that’s always on my mind
I have a LOT of partially-written blog posts, but am struggling to get any of them finished (obviously). Much of the problem is that they have so many dependencies on each other. Clearly, then, I should consider refactoring my writing plans. 🙂
So let’s start with this. Here, in no particular order, is a list of some things that I’ve said in the past, and which I still think are or should be of interest today. It’s meant to be background for numerous posts I write in the near future, and indeed a few hooks for such posts are included below.
1. Data(base) management technology is progressing pretty much as I expected.
- Vendors generally recognize that maturing a data store is an important, many-years-long process.
- Multiple kinds of data model are viable …
- … but it’s usually helpful to be able to do some kind of JOIN.
- To deal with the variety of hardware/network/storage arrangements out there, layering/tiering is on the rise. (An amazing number of vendors each seem to think they invented the idea.)
2. Rightly or wrongly, enterprises are often quite sloppy about analytic accuracy.
- My two central examples have long been inaccurate metrics and false-positive alerts.
- In predictive analytics, it’s straightforward to quantify how much additional value you’re leaving on the table with your imperfect accuracy.
- Enterprise search and other text technologies are still often terrible.
- After years of “real-time” overhype, organizations have seemingly swung to under-valuing real-time analytics.
| Categories: Data models and architecture, Database diversity, Predictive modeling and advanced analytics, Public policy, Theory and architecture | 1 Comment |
Some things I think about politics
When one tries to think comprehensively about politics these days, it quickly gets overwhelming. But I think I’ve got some pieces of the puzzle figured out. Here they are in extremely summarized form. I’ll flesh them out later as seems to make sense.
1. Most of what people are saying about modern tribalism is correct. But partisanship is not as absolute as some fear. In particular:
- There are populist concerns on the right and left alike.
- Partisans of all sides can be concerned about privacy, surveillance and government overreach.
2. The threat from Trump and his Republican enablers is indeed as bad as people fear. He’s a major danger to do terrible, irreversible harm to the US and the rest of the world. To date the irreversible damage hasn’t been all that terrible, but if Trump and his enablers are given enough time, the oldest modern democracy will be no more.
All common interests notwithstanding, beating Trump’s supporters at the polls is of paramount importance.
3. I agree with those who claim that many of our problems stem from the shredding of trust. But few people seem to realize just how many different aspects of “trust” there are, nor how many degrees there can be of trustworthiness. It’s not just a binary choice between “honest servant of the people” and “lying, cheating crook”.
These observations have strong analogies in IT. What does it mean for a system to be “reliable” or to produce “accurate” results? There are many possible answers, each reasonable in different contexts.
| Categories: Public policy, Surveillance and privacy | 2 Comments |
Politics can be overwhelming
Like many people, I’ve been shocked and saddened by recent political developments. What I’ve done about it includes (but is not limited to):
- Vented, ranted and so on. That’s somewhat therapeutic, and also let me engage the other side and try to understand a little better how they think.
- Tried to understand what’s happening. I probably have had more available time to do that than most people. I also have a variety of relevant experiences to bring to bear.
- Neglected my work somewhat while doing all that. This neglect has now stopped. After all, the future is quite uncertain, so we should probably work hard in the present while business is still good.
- Written up some of what I’ve figured out. Of course. That’s what I do. But it’s only “some”, because … well, the entirety of politics is overwhelming.
- Tried to find specific, actionable ways to help. Stay tuned for more on that part.
As for those writings: Read more
| Categories: Public policy, Surveillance and privacy | 1 Comment |
The chaotic politics of privacy
Almost nobody pays attention to the real issues in privacy and surveillance. That’s gotten only slightly better over the decade that I’ve written about the subject. But the problems with privacy/surveillance politics run yet deeper than that.
Worldwide
The politics of privacy and surveillance are confused, in many countries around the world. This is hardly surprising. After all:
- Privacy involves complex technological issues. Few governments understand those well.
- Privacy also involves complex business issues. Few governments understand those well either.
- Citizen understanding of these issues is no better.
Technical cluelessness isn’t the only problem. Privacy issues are commonly framed in terms of civil liberties, national security, law enforcement and/or general national sovereignty. And these categories are inherently confusing, in that:
- Opinions about them often cross standard partisan lines.
- Different countries take very different approaches, especially in the “civil liberties” area.
- These categories are rife with questionably-founded fears, such as supposed threats from terrorism, child pornographers, or “foreign interference”.
Data sovereignty regulations — which are quite a big part of privacy law — get their own extra bit of confusion, because of the various purposes they can serve. Chief among these are: Read more
| Categories: Surveillance and privacy | 2 Comments |
The technology industry is under broad political attack
I apologize for posting a December downer, but this needs to be said.
The technology industry is under attack:
- From politicians and political pundits …
- … especially from “populists” and/or the political right …
- … in the United States and other countries.
These attacks:
- Are in some cases specific to internet companies such as Google and Facebook.
- In some cases threaten the tech industry more broadly.
- Are in some cases part of general attacks on the educated/ professional/“globalist”/”coastal” “elites”.
You’ve surely noticed some of these attacks. But you may not have noticed just how many different attacks and criticisms there are, on multiple levels.
| Categories: Amazon and its cloud, Facebook, Google, Public policy | 3 Comments |
Notes on artificial intelligence, December 2017
Most of my comments about artificial intelligence in December, 2015 still hold true. But there are a few points I’d like to add, reiterate or amplify.
1. As I wrote back then in a post about the connection between machine learning and the rest of AI,
It is my opinion that most things called “intelligence” — natural and artificial alike — have a great deal to do with pattern recognition and response.
2. Accordingly, it can be reasonable to equate machine learning and AI.
- AI based on machine learning frequently works, on more than a toy level. (Examples: Various projects by Google)
- AI based on knowledge representation usually doesn’t. (Examples: IBM Watson, 1980s expert systems)
- “AI” can be the sexier marketing or fund-raising term.
3. Similarly, it can be reasonable to equate AI and pattern recognition. Glitzy applications of AI include:
- Understanding or translation of language (written or spoken as the case may be).
- Machine vision or autonomous vehicles.
- Facial recognition.
- Disease diagnosis via radiology interpretation.
4. The importance of AI and of recent AI advances differs greatly according to application or data category. Read more