You are familiar with the 4th Amendment? These acts are a clear violation of 4th Amendment rights, rights which extend to both citizens and non-citizens.
>When do ICE agents need a warrant to arrest immigrants?
>A judicial warrant is a legal order authorizing law enforcement’s search, seizure or arrest on private property. Judicial warrants are signed by a judge.
>Immigration agents also use administrative warrants, which carry lower legal weight. Administrative warrants are signed by federal agents such as immigration judges or officers. These warrants allow ICE agents to arrest someone in public places. However, they don’t give officers the right to enter private property.
>Although ICE agents are required to have a judicial warrant to enter a person’s home, they are not required to have a judicial warrant to arrest someone in public spaces, such as the immigration court building.
>"Lander is incorrect that a judicial warrant is required," Aaron Reichlin-Melnick, a senior fellow at the American Immigration Council, an immigrant-rights advocacy group, said on X.
>An administrative warrant isn’t always required to arrest someone in public. According to immigration law, agents can arrest an immigrant without a warrant if they have "reason to believe" the immigrant is in the U.S. without authorization and "is likely to escape before a warrant can be obtained for his arrest."
>ICE agents generally can’t arrest U.S. citizens, because they aren’t committing a civil immigration violation. However, an agent may arrest a U.S. citizen on the grounds that they believe the person is in the U.S. illegally. The person would be released after showing proof of citizenship.
>However, Lander wasn’t arrested on immigration grounds, said Alexandra Lopez, a Chicago-based immigration attorney. The agent accused Lander of obstruction.
>"In this scenario they are acting as federal law enforcement agents who are arresting a U.S. citizen on criminal, not immigration, grounds," Lopez said. "ICE claims they were detaining Comptroller Lander in their capacity as federal law enforcement agents, not immigration enforcement agents."
Immigration law is complicated.
I'm not some right-wing nutter. I'm just a lefty that thinks we're definitely shooting ourselves in the foot by really misunderstanding what's actually happening. Nullification of immigration laws is, in fact, a right that states can exercise, but it's overt nullification is absolutely an escalation that undermines public trust because it force the feds to send enforcement officers into a hostile area.
We should fight to win the immigration debate with persuasion, in the legislature. We need to have the law on our side, and we need to have the populace on our side. Right now, we have neither. We're operating a nullification campaign, and unlike the successes of legalizing marijuana, we're losing this one. If we want to keep doing this, that's fine, but I don't want people out there pretending that lawful detentions are kidnappings. It's dumb, it's a bad look, and it kind of doesn't care about the complexities of the predicament we're in.
This is a forum for nerds. I expect people to actually be able to google this shit.
>Cities like Milwaukee require police officers to make their names or officer identification numbers visible. This ensures that if there is an allegation of wrongdoing, the officer can be identified. This also is to guard against impersonators.
>There are exceptions. For instance, Milwaukee police detectives wear "plain clothes," often a dress shirt and pants. And, of course, undercover officers dress in such a way not to be identifiable, by design.
>At the 2024 Republican National Convention, where 4,500 outside officers came to assist, the Milwaukee Police Department was clear that any visible uniform change would be deemed an escalation of force.
>Federal law enforcement, like FBI and ICE, for the most part do not have an official uniform, though during raids they typically wear body armor, windbreakers or other gear with the name of their agency emblazoned on it.
>At times, federal and local law enforcement have covered their faces during raids, most often when they involve gangs or terrorism where there is a risk of retaliation.
>In 2025, ICE officers have increasingly been wearing face coverings. ICE leaders said that's because their officers increasingly are being assaulted and harassed online.
I agree with you that ICE agents should absolutely show their faces. That said, it's not unprecedented. I also think it's naive to think there would not be retaliation against them personally.
>The requirement for police officers to provide their name and badge number varies across the United States. While no federal law mandates disclosure, many states and municipalities have their own statutes aimed at enhancing transparency and accountability. These laws often require officers to identify themselves during specific interactions, such as traffic stops or arrests, to ensure citizens can hold law enforcement accountable.
ICE where uniforms that say "ICE" in big letters. That's identification. Undercover police officers might identify themselves during an arrest, but only as "police." Undercover police officers aren't going to give you their name and badge number if you ask them.
>Situations Where Disclosure May Be Withheld
>While officers are generally expected to provide their name and badge number, there are situations where disclosure may justifiably be withheld. During undercover operations, revealing an officer’s identity could compromise safety and the operation’s integrity.
>In protests or crowd control situations, officers may face security concerns, such as risks of doxxing or harassment. To address this, some departments allow officers to withhold identification while still requiring visible markers, like badge numbers, to maintain accountability without endangering safety.
>I agree with you that ICE agents should absolutely show their faces. That said, it's not unprecedented. I also think it's naive to think there would not be retaliation against them personally.
And if that occurs, whoever is responsible should be prosecuted.
You know that whole "rule of law" thing that seems to be so unfashionable, among certain masked folks and the liars who run them, these days?
You are making a legalistic argument to justify absolutely monstrous behavior and you should probably spend some time examining why you are doing that. If the law justifies an atrocity, you should not defend the law.
I'm not someone who thinks all laws should be followed blindly. When human rights are on the line, yes, we have an obligation to resist. Some people think enforcing immigration law is a violation of human rights.
I do not think it is a reasonable position to consider deportaion of folks overstaying visas as "a violation of human rights" in the vast majority of cases. Where we are breaking up families with young children is where I would draw my line, and that is certainly happening, but again my concern here is with the escalation that is nullification.
I simply think that if I were to go to, say, the UK and decided to not board a flight home and make a life for myself that I could be forcibly deported... and the Labour Gov't in the UK does forcible deportations:
I think the ideal solution is to create a system where overstaying a visa is practically impossible. This way people could not find them in a situation where they've established a life that would make leaving especially painful. However, since it has proven to be too practically difficult to negotiate comprehensive immigration reform for various reasons, the American left -- a left that I consider myself a part of -- has gone in the complete opposite direction for most of my lifetime. We have established an overt nullification policy that effectively facilitates folks ignoring immigration law. Now we have to deal with immigration enforcement we don't like, and it will be very difficult for us to protect young children losing a parent because we've decided that we want to effectively facilitate all folks here illegally, not just those who have found themselves with young families.
> I do not think it is a reasonable position to consider deportaion of folks overstaying visas as "a violation of human rights" in the vast majority of cases.
This is a motte/bailey. Deporting people is not inherently a violation of human rights. However, when judges have to clarify that "detainees" must be provided water and toilets[0], I think it's pretty clear that their human rights are being violated. The significant objection is to that, not to any semblance of immigration enforcement.
> I think the ideal solution is to create a system where overstaying a visa is practically impossible.
I can assure you that you do not want this, it is predicated on a level of government invasiveness that would be unpalatable to both citizens and legal immigrants. Some abuse is the cost of many well functioning systems.
> However, since it has proven to be too practically difficult to negotiate comprehensive immigration reform for various reasons, the American left -- a left that I consider myself a part of -- has gone in the complete opposite direction for most of my lifetime. We have established an overt nullification policy that effectively facilitates folks ignoring immigration law.
It is somewhere between deeply misinformed and rhetorical malpractice to say this, pretending that the American right bears no responsibility for preventing progress on immigration reform and that there haven't been multiple attempts by the left to improve things here that were blocked by the right (including multiple iterations of DREAM and various attempts at asylum reform).
I mean, I disagree with you on all counts. I think it’s always fair to care about treatment of detainees… we don’t use the repeated inflammatory “kidnappings” of our concern is merely detainees treatment.
Other states, such as the UK, make it obscenely difficult to exist without documentation. They certainly do not tacitly endorse it. To suggest “I wouldn’t like” policies that plenty of western countries engage in seems naive.
Finally, the Republicans temperament on legal immigration is horrific, but they are in the position to ignore attempts to change the law because the law is on their side… like any issue in democracy, that means the Democrats are the party that needs to change minds.
> we don’t use the repeated inflammatory “kidnappings” of our concern is merely detainees treatment.
No, but we do use it for otherwise unlawful stops without probable cause that lead to people being put in detention facilities that don't have water or food.
> like any issue in democracy, that means the Democrats are the party that needs to change minds.
This is not the argument you just made. You were (and are) arguing for collaboration. That's not "changing minds". In my opinion, being loud and not collaborating with federal forces, to make them engage in violence themselves is very effective at changing minds, as we see with cratering public support for these kinds of things.
I admit I can't quite follow what your philosophy seems to be here, at best I could summarize what I've seen as "Republican immigration policy is bad and has grown more unconscionable but I actively support it because Democrats didn't fix it already", but that seems weird.
I wouldn’t put it in those terms, but I think I understand you point and yes, the general point is that I think we should enforce laws we don’t like unless they directly run up against what we see as a serious violation of human rights. I think that is generally a good idea, because it preserves a governmental structure we all generally agree with: something approximating one person one vote for representation, with a few caveats thrown in.
Democracy falls apart rapidly if your strategy is to only enforce laws you endorse. Democracies that fall apart are typically replaced with undemocratic systems. On top of that, civil conflict is horrible for human flourishing, so shit needs to get really, really bad before that discussion happens. I see this as a very strange sword for the American left to fall on.
The counterargument to this is pretty straightforward: what is being done in Midway blitz isn't democratic and is bordering on autocratic. We have a responsibility not to normalize and acquiesce to a transition to an undemocratic system.
Keep in mind these laws weren't enforced in this way for the past 50 years. It's difficult to accept that this was just democratic party disinterest in enforcing them. It really seems like no one wanted to.
>what is being done in Midway blitz isn't democratic and is bordering on autocratic
I mean, we're talking about a democratically elected government enforcing democratically decided laws. I understand your sentiment, and generally agree with you that it "feels" that way, but I think there is zero substance to that claim considering the entire process of how we got here is democratic. I don't like it, but here we are.
>Keep in mind these laws weren't enforced in this way for the past 50 years.
We're obviously not going to see eye to eye on this. Illegal immigration is very obviously a major concern for a huge portion of the electorate, and because of the significant polarization on the subject, nullification here is going to lead to conflict as long as the federal electorate wants to enforce those laws. I obviously think this situation is unfortunate. I'm incredibly supportive of massively expanding American immigration, but it's difficult for me to get on board with nullification.
You're doing a motte and bailey again. I, at least, don't object to some level of immigration enforcement.
What people do seem to object to, and what is unprecedented, is the aggression of enforcement, with roving packs of CBP officials going on snatch-and-grabs in random cities and detaining anyone who is latino-looking, including some citizens. That isn't how immigration law has been enforced over the past 5 decades. It's new. It wasn't policy under Bush or Obama or Biden or even under Trump the first time. The laws were not enforced like this since WWII.
The last time the Alien Enemies Act was invoked was during WWII. Its use this year was only lawful if you agree with the interpretation that certain Presidential determinations are wholly unreviewable by courts, an interpretation that so far, courts (including SCOTUS) have been unwilling to agree to.
There is significant controversy over whether much of this is even legal at all. And yet you seem to be of the opinion that state and local governments have some kind of responsibility to assist with actions they believe are illegal overreach. Because you're framing a lack of active participation as "nullification". You at least see why that's odd, right?
I've said over and over that nullification is the right of states and municipalities. My entire point is that it's inherently an escalation. When the feds choose to enforce a law is areas that are actively trying to prevent that law from being enforced, almost by definition requires a heightened level of conflict in how that enforcement is done.
I agree with you that this is "novel" but the idea is that this isn't a pendulum that swings back an forth. It's a cascade where the dam is breaking, and when it does, creates a wildly different paradigm than existed previously.
I don't like what is happening. I can just see why it's happening, and understand and appreciated the justifications for it.
Two this, first, I want to jump back to something you said earlier:
> Nullification of immigration laws is, in fact, a right that states can exercise, but it's overt nullification is absolutely an escalation that undermines public trust because it force the feds to send enforcement officers into a hostile area.
Do you see why this might actually be seen as increasing public trust in local LEOs who aren't participating in human rights abuses?
> We should fight to win the immigration debate with persuasion, in the legislature. We need to have the law on our side, and we need to have the populace on our side.
And can you see why not condoning those abuses gets the populace on "our" side?
Second, you have asserted something like
> When the feds choose to enforce a law is areas that are actively trying to prevent that law from being enforced
a few times now. And I'd like you to clarify: in January 2025, what actions was Chicago taking that were "actively preventing [immigration] law from being enforced"? And what actions do you see municipalities engaging in today that are "actively trying to prevent [immigration] law from being enforced"?
And if you were in charge, what would you do instead? Keep in mind, as a mayor or police captain or whatever, you cannot tell Greg Bovino what to do. You can assist him, but his use of force policies are different than yours, and you cannot make him or his officers follow your directives.
>Do you see why this might actually be seen as increasing public trust in local LEOs who aren't participating in human rights abuses?
> And can you see why not condoning those abuses gets the populace on "our" side?
I’m not sure how this is relevant. I’ve repeatedly noted my concerns with some of the enforcement. My only point is that nullification — effectively by definition — raises the stakes for potential conflicts.
> in January 2025, what actions was Chicago taking that were "actively preventing [immigration] law from being enforced"? And what actions do you see municipalities engaging in today that are "actively trying to prevent [immigration] law from being enforced"?
I mean, I think sanctuary city laws are clearly problematic. I obviously appreciate the benefits that accrue in the short term, and it’s an odd equation when approaching the problem from a shot vs long term perspective when it comes to harm reduction, but we’ve clearly gotten to the point where the general population wants something done that is incompatible with maintaining those policies. Yes, there are trade offs. We very rarely offer the same luxury to other violations.
> And if you were in charge, what would you do instead? Keep in mind, as a mayor or police captain or whatever, you cannot tell Greg Bovino what to do. You can assist him, but his use of force policies are different than yours, and you cannot make him or his officers follow your directives.
If I were in charge, I would have been voted out of office long ago. The fundamental problem here is two political sovereigns in a fistfight.
But suppose I were somehow in charge of the state govt, the first thing I would do is what Scott wiener did in CA, and pass state laws requiring all law enforcement to show their faces in my state. The feds have authority on immigration, but they don’t have immunity to state laws where the 10th amendment applies.
If I were the mayor, yes, I would be asking the police to assist in enforcement wherever they can, with their cameras on, recording everything.
> I mean, we're talking about a democratically elected government enforcing democratically decided laws.
No, we are talking about a government repeatedly, flagrantly, breaking the law, and then lying about it and repeatedly getting caught, by courts, by video, etc.
That’s a very important concern that is not directly relevant to whether or not the laws being enforced are democratic or whether the person who is enforcing those laws is a democratically elected representative.
I’ve repeatedly noted my concerns and problems with many of the actual enforcement. That said, there is an ocean of difference between having unjust laws and unjust policing.
> That’s a very important concern that is not directly relevant to whether or not the laws being enforced
Well, no, its relevant to whether what is happening is arbitrary and unlawful use of force or enforcement of the law, and if it is the former, then the whole question of "whether or not the laws being enforced are democratic" is misguided, because the shared premise assumed by both options presented is false.
Your concern is with the executive powers, that's an issue for the judicial branch. Our discussion is about the legislative branch and the electoral process. That's how we create law. This discussion is about the consequences of nullification of law that we don't like that does not violate some inherent human right (and no, I think it's fairly absurd to suggest that overstaying a visa is some human right, and I also think it's even difficult to suggest that seeking asylum in one nation specifically, and not a neutral third-party nation is some human right).
You're talking about the specific day-to-day of enforcement, which I've repeatedly said I don't like, and is probably a problem in many cases. That is important, it's also worth discussing, it's just very much NOT relevant to a discussion of the risks and consequences of states going down a path of nullification of a law that is popular on the federal level.
You keep saying “nullification”. Can you explain precisely what you mean by that?
Because as far as I’m aware, immigration law is not a concern of the state, and what folks typically mean when they say “nullification” in this context is “the state isn’t doing the fed’s job for them.”
You also brought up warrants to enter private property. What do you make of the incident a few days ago where an agent hopped a fence to arrest someone, without a warrant? Should we just ignore those violations of our rights?
>Because as far as I’m aware, immigration law is not a concern of the state, and what folks typically mean when they say “nullification” in this context is “the state isn’t doing the fed’s job for them.”
It's not just immigration law, it's any federal law. States have the right to ignore federal law if they like. This is called nullification. However, it very, very rarely happens because its inherently undemocratic. It especially rarely happens to the extent that cities and states pass explicit laws that order state law enforcement to ignore federal laws, and even work against the federal government's interests.
It's happened recently with marijuana legalization, with success. Where the federal government did some raids, but marijuana legalization is politically popular, so they backed off... and there has even been talk in some years of ending the illegality of marijuana federally.
State nullification has been somewhat unsuccessful with illegal immigration. These raids are the result of the federal government going its own way to enforce the law without cooperation of the states. The last time we saw this level of federal enforcement against state objection is after Brown v Board of Education: https://en.wikipedia.org/wiki/Little_Rock_Nine
I good comparison to the seriousness of nullification as an act that is inherently an escalation is gun control laws. Suppose some red states wanted to just nullify the National Firearms Act -- https://en.wikipedia.org/wiki/National_Firearms_Act -- The are perfectly in their rights to ignore federal laws and allow firearms dealers to sell unregistered, suppressed, machine guns to felons. The only way neighboring blues states -- obviously outraged that this is happening -- can do anything about this is by seeking federal enforcement, again, which would include raids, arrests, etc.
>You also brought up warrants to enter private property. What do you make of the incident a few days ago where an agent hopped a fence to arrest someone, without a warrant? Should we just ignore those violations of our rights?
I'm very much not saying ICE is always acting within the law. Like any other policing force, they're going to make mistakes (intentional or otherwise). We should be very angry about those things, especially if they're happening in bad faith. The problem I see is that when we're yelling about actually -- and unfortunately -- legal things then those serious issues are just going to look like background noise. The other serious problem is that all this crying wold literally makes the left look undemocratic. You don't like the law? Fight to change it. Don't just take the ball and go home, and then cry when the neighbors come to your house to get the ball back.
There is a world of difference between “passing a state law that directly contradicts federal law” and “declining to proactively enforce federal laws in ways that are not required by those laws.”
To drive the point home: federal immigration laws are already enforced by federal agencies. Here in IL, state and local officials cooperate to the extent required by law. There are no federal laws on the books requiring them to do the job of the federal government for them (they could pass one, but they haven’t).
Calling that “nullification” is intellectually dishonest. As you said - “if you don’t like the law, fight to change it.” Don’t pretend it’s something it’s not.
>Here in IL, state and local officials cooperate to the extent required by law.
This is clearly false in regards to most federal laws. To illustrate this, I'll take an exceptional example. If there where a serial killer who was living in IL, but had only killed anyone in other states, I suspect that IL government would likely go out of their way to assist the Feds in apprehending this killer, even though this is not required by state law.
IL would likely do the same for many, if not most, federal laws. The point of nullification is exactly when the state does not help when asked, still there are reasons for practical resources there, but it becomes very obvious nullification when the state passes laws preventing individuals who would LIKE to help, like local policed departments, from helping even if they wanted to. And this is exactly what has happened in many blue states.
Pretending that's not overt nullification is unserious.
Not assisting with enforcement acts you don't feel are worthwhile is not nullification. I'm not engaging in "nullification" when I don't call the police on a jaywalker. Or I mean maybe you think this is, but then police engage in wildcat strikes all the time, or change enforcement priorities, or whatever you want to frame it as. Calling a difference in prioritization "nullification" wrong, especially if local police in immigrant communities want to maintain good relationships with those communities. I think it's laudable that some police forces show an interest in serving their communities interests, as opposed to yearning to be fashy.
> but it becomes very obvious nullification when the state passes laws preventing individuals who would LIKE to help, like local policed departments, from helping even if they wanted to. And this is exactly what has happened in many blue states.
Can you give examples?
Keep in mind, "sanctuary city" policies are usually actually supported by local police forces, because while they may look not tough on crime (and for this reason sometimes police forces halfheartedly lobby against them), they actually make on-the-ground local policing easier, because they engender trust between the local police force and immigrant communities who otherwise might not report crimes at all.
>I’m not going to engage with you if you’re going to get in multiple threads and refer to things as “fashy.”
>It’s difficult enough to engage in a heterodox view in good faith. I don’t need to deal with slapdash bullshit.
I see we've reached the point in the discussion where you 'abruptly fall silent, loftily indicating...that the time for argument is over.'
Good fascist! Nice fascist! Late for a Bund meeting, are we?
Source:
“Never believe that anti-Semites [or in this case, fascist apologists] are completely unaware of the absurdity of their replies. They know that their remarks are frivolous, open to challenge. But they are amusing themselves, for it is their adversary who is obliged to use words responsibly, since he believes in words. The anti-Semites have the right to play. They even like to play with discourse for, by giving ridiculous reasons, they discredit the seriousness of their interlocutors. They delight in acting in bad faith, since they seek not to persuade by sound argument but to intimidate and disconcert. If you press them too closely, they will abruptly fall silent, loftily indicating by some phrase that the time for argument is past.” ― Jean-Paul Sartre[0]
I am sorry but you're delusional if you think any of that is happening and they're acting in a legal manner. A small sample of the links in that post show ICE are actively violating constitutional rights and flaunting the rule of law. This same organization is actively ignoring a federal judge's orders to not use crowd control weapons on people who pose no threat.
Very open - Python is one of the better communities out there for openness and inclusion. The pip folks are lovely people and would welcome your help - https://pip.pypa.io/en/latest/development/.
You can alert via a number of mechanisms: email, PagerDuty, Slack et al, etc (I talk about most of those in Chapter 9).
I have never been very keen on alerting dashboards, I find they are rarely actually reviewed and flash red for days or weeks. :) So I only covered metrics/graphing as a console rather than a status console. If you want to add such a console it'd be easy to output Riemann events via an API to such a console.
It's a spectrum to me. We're way behind the curve on monitoring and the "state of the art" in, anywhere but cutting edge shops, is woeful. I'd love folks to be able to anomaly detection easily and simply but the technology and tools aren't quite there yet. I am just hoping to get folks to advance their environments a little way forward.
Yes, the situation is horrible. I'm reluctant to believe that the "cutting edge shops" are doing very well.
For good "tools", I have a good paper on the subject, but from all I can see there is essentially no interest. People would prefer not to be bothered. The attitude seems to be, if there is a problem, then we will detect it, eventually if not soon, and then we will fix it.
It was published, in the Elsevier journal 'Information Sciences' in 1999.
It appears to be the first, and the first large, collection of statistical hypothesis tests that are both multi-dimensional and distribution-free.
I try to be anonymous here at HN, but I'm willing enough to send a PDF of the paper to anyone who wants a copy. E.g., ask for a copy and leave your e-mail address on your HN profile, at least temporarily.
The main point of the paper is that we do get an hypothesis test. In particular, we get to select false alarm rate and then get that rate essentially exactly in practice.
It's behavioral monitoring -- it assumes that the past and future of healthy performance are, to be simple, statistically the same. So, right, its for a server farm or network that is statistically relatively stable, that is statistically unchanging, in what it is doing. The site can be wild and crazy, but it has to continue to be wild and crazy in statistically the same way.
In particular, the work is for detecting zero day problems, that is, problems never seen before. Maybe the philosophy here is that when we get a new problem and detect it, then we fix the cause of the problem and never see it again and, then, again are left looking for zero day problems.
Then the work uses past data -- hypothesis tests have done that since Karl Pearson 100+ years ago, and now parts of computer science do something similar and call it training data in unsupervised learning or some such. The approaches of just statistical hypothesis testing make more sense to me.
The key, core mathematical argument is a finite algebraic group of measure preserving transformations on the data. I believe that there are connections with U-statistics, e.g., as in an advanced statistics book by Serflng.
This stuff with groups and measure preserving is a little like some classic arguments in ergodic theory. On the page, the math looks awful, but actually it is conceptually quite simple.
But, you don't need to dig into the math too much.
For the actual calculations, those are based on nearest neighbors (although other options also work with the basic math). At least since the paper, others have thought of using nearest neighbors, but they didn't have an hypothesis test because they didn't know how to calculate and adjust false alarm rate. So, they have an heuristic instead of an hypothesis test. So, again, the main contribution of the paper is that it really is an hypothesis test, that is, know and get to adjust false alarm rate (conditioned on the old data and also true in long run expectation over the conditioned work -- standard result in conditional expectation from the Radon-Nikodym theorem in measure theory).
For detection rate, there is some good news, not as good as from the classic Neyman-Pearson result (in practice in the context we don't have enough data to do much with Neyman-Person), but nice: In a useful sense, for the selected false alarm rate, the work gives the highest possible detection rate. Really the mathematical key here is just Fubini's theorem (the measure theory version of interchange of order of integration). Intuitively, the technique has the largest area where alarms are raised consistent with the selected false alarm rate.
For the practical application, do need some help with some computational geometry. For that, I dreamed up some work. Soon I found that part of what I dreamed up was k-D trees, e.g., as in Sedgewick's book on algorithms. But there is more -- need some cutting planes. I programmed most of it 20+ years ago in PL/I but finally dropped it due to lack of interest.
I have some ideas for more results of interest and more papers, but after 20+ years of no interest I just gave up.
More can be said, but I stopped the research when I discovered, about 20 years ago, that no one was interested. The paper was published in 1999, and since then interest has been quieter than the tombs of ancient Egypt. So, I'm doing a startup that is quite different.
I dreamed up the work when I saw the need, or at least as I regarded the need, way back in about 1990 when I was in an AI group at the IBM Watson lab doing work on monitoring and management of large server farms and networks. The AI work was trying to build on essentially just threshold detectors. There was no attention to false alarm rate or a best detector -- highest detection rate for given false alarm rate. The classic Neyman-Pearson result was ignored. I was our guy with GM Research, and we gave a paper at the Stanford AAAI IAAI conference. But I was outraged by the lack of concern for false alarm rate, ignoring hypothesis tests and distribution free hypothesis tests (long common in the social sciences), and with no attention at all to multi-dimensional data.
The real world context is just awash in multi-dimensional data. Treating the data components separately in effect says that the geometrical region of healthy behavior is just a box. Bummer. Box too small -- get false alarm rate too large. Box too big, get too many missed detections. Problem: A box is a poor fit to reality. Simple stuff.
How to see this? Monitor CPU busy and page faults per second and look for anomalies, e.g., thrashing, a program allocating infinite memory, etc. Then the normal behavior is just a 2-D box? I don't think so! But, sure, need to automate picking the shape of the region of healthy behavior.
For the distribution-free stuff, that is where we make no assumptions about probability distributions. I got a kick in the back side on that sitting one day in the office of Ulf Grenander, one of the world's best ever statisticians, at Brown (I got accepted to grad school there; was considering going; went elsewhere instead). Grenander had been looking at computer performance data and was shocked at how different it was from the data, e.g., biomedical, he had been used to. So, right, Gaussian assumptions and more go out the window!
So, really, just want to make no assumptions about distributions, want to be distribution-free (a.k.a., non-parametric although I believe distribution-free is more appropriate terminology).
For multi-dimensional, at IBM I got a slap in the face: There was a cluster of computers doing transaction processing. There was some front end load leveling that sent the next transaction to the least busy computer in the cluster. Okay. But one day one of the computers got sick, just a little sick in the head, and was doing a very silly thing -- it was throwing all its incoming transaction work into the bit bucket! Thus, this computer looked to the load leveling as not very busy and, thus, was getting nearly all the transactions. Thus, nearly all the transactions for the whole cluster were going into the bit bucket. Bummer.
So, I thought, to detect this anomaly, want somehow to look at all of the computers in the cluster at the same time and compare them with each other, that is, have all the data in some appropriate region in some space of several dimensions, a region that works whether the cluster is busy or not. So, want to be multi-dimensional, that is, don't want just threshold detectors on variables one at a time.
There are more war stories where the importance of being multi-dimensional is crucial. Really, commonly separating multi-dimensional data into its components and treating the components separately can be throwing away a lot of crucial information which stands to give a poor combination of false alarm rate and detection rate.
Heck, in principle the region of healthy performance can be a fractal, say, like the Mandelbrot set, and, so, somehow we need to approximate that. Can we do that? Basically with nearest neighbor, or k-nearest neighbors (which also works), yes.
There is now a good opportunity for my work: My work can use a LOT of training data, and in the near real-time detection work can want to do a lot with that data. So, could use fast access to a lot of data which doesn't change very fast. So, sure, use some big solid state disks (SSDs)! A few of the Samsung 14 TB drives should do wonders for my paper!
My view is, anyone doing monitoring of a large server farm or network and not using what is in my paper is not being fully serious. And, since get to adjust false alarm rate, say, to one a month, can't say that can't afford the extra false alarms.
Uh, I left out: For each alarm, get told the lowest false alarm rate at which the real-time input data is still an alarm -- so get an indication of alarm seriousness.
More is possible, but at least have to be using what I cooked up in 1990, wrote prototype software for in the early 1990s, and published in 1999.
I did the work a long time ago, guys! And since then, there have been various serious consequences from anomalies, intrusions, etc. Maybe in some of those cases, my work would have done good, early detection. My work looks a heck of a lot better than anything else!
So I think the mathematics to make this work is not the problem. How do you engineer this though?
If I were to try and build a platform that could do this in real-time for, lets say, a million metrics per minute, can you engineer something that would scale horizontally to do this? Can it be done by cobbling together various open-source tools/libraries currently out there? Then how would you present the results in a way that someone that's not necessarily "mathematically inclined", say for example, your typical operational support person, that they could meaningfully interpret whatever your system is spitting out?
That's for me the hard part, is to get those two components working well. Make it scale, make it idiot friendly. If you can't get those parts right, it doesn't matter what you're trying to do.
I say this because I've spent the last 6 years in the application performance management space and "the best" way to handle alarms at the moment is to put down a team, literally a team, of people and have them hand-tune thresholds by looking at a combination of history, incidents/outages and root cause outcomes, domain specialist inputs (like DBAs or application server specialists). You send out a false or noisy alarm to an ops guy too many times and they become desensitized. You don't put enough context in your alarm messages, they won't use it (logging into a tool is asking too much, the email must contain everything they need or they complain).
Any form of dynamic baselining is just too noisy. The simplest example is trying to "baseline" CPU usage. CPU usage without something trivial like comparing to run-queue is stupid. It's actually even more stupid because you should be looking at things top-down, i.e. so what if the CPU is 100% and the run queue is 100, are any user facing transactions slowing down? i.e. is there customer impact. It could be some batch job kicking off. So in short, anything that looks at a metric in isolation is stupid, dynamic baselines with time of day, day of month, etc. it's all rubbish shit, you're wasting your time with this approach. This is the sad state that current "cutting edge" third generation APM tools offer though.
The same as for several others here at HN, leave your e-mail address in
your HN profile, and I will try to remember to return, use your e-mail, and send you a PDF of the paper, 1999 in Information Sciences.
Scaling? Can scale about as much as you want. Get some racks of high end servers and a fast server farm LAN, collect the data with whatever instrumentation, and let it run.
For false alarm rate, just shovel in the training data, pick a false alarm rate, say, one a week, one a month, and let it run. Don't hand tune anything. In effect, the hand tuning is replaced by the adjustable false alarm rate and where you know the rate in advance and, with the statistical assumptions, get that rate exactly in practice. Statistical hypothesis testing has had adjustable false alarm rates for 100+ years. Sorry that computer monitoring has been struggling with that.
Yes, you will need to make a judgment call about the assumption that the system now is statistically the same as during the past, say, three months of training or historical data of apparently healthy behavior.
For rates of false alarms, rates of missed detections of real problems -- the server farm bridge staff and the network operations center (NOC) can understand those. I was invited for a free lunch and gave a presentation to the operations staff at the main NASDAQ facility in Trumbull, CT, and the operations staff, maybe 30 people, understood the basics right away.
For the math, the only tricky part, and the core of my paper, is how the heck to know and adjust the false alarm rate, but operationally for the staff that is just trivially easy.
Once I was at Morgan Stanley and talking to their main Unix system administrator. He'd just come from a meeting on how the heck to monitor his Unix systems, and I explained my work. He nearly jumped up and exclaimed "We can use this right away!". But they didn't give me an offer, ask me to consult (my paper was not published yet), etc. So, really, they didn't much care.
For how to report the alarms, I'd guess feed into some standard system management infrastructure, consoles, whatever from HP, CA, EMC, Microsoft, etc. There's a way to get a real time, running strip chart that says what the false alarm rate would be for the data just observed to be an alarm. If people want to watch 500 of those, okay by me. But mostly would want a way to display the strip chart for detectors that just gave alarms or, given an alarm for, say, a Cisco switch, display the strip charts for all the monitors of that switch -- for insight and an aid to diagnosis.
> The simplest example is trying to "baseline" CPU usage. CPU usage without something trivial like comparing to run-queue is stupid.
Of course it is. My work would still give the selected rate of false alarms, but the detector would likely also have poor detection rate. I.e., with just CPU usage, the poor detector just doesn't have enough information to do anything very good.
Now THAT'S in part why want to be multi-dimensional. So, say, feed in PAIRS of CPU busy and run-queue length. Maybe include some more variables, e.g., time of day.
So, here's your first judgment call: What to monitor and what variables to combine to several variables to feed to some one detector. It's clear you have some insight. Good. In time there will be some good ideas for what variables to use to monitor a Cisco switch, an Oracle database, a Windows Server, etc.
I have in mind some more research to help make such selections, but, again, for 20+ years no one was interested. You described the problems well, and I made good progress on solving them, but, still, no one was interested. No one. Did I mention, no one? The paper was right there in a peer-reviewed journal, and it was treated like a source of leprosy. Not my fault. And this is not nearly the first time I've publicized this on HN.
Really, there's hardly a well known VC firm that hasn't heard from me. And there's hardly a one I ever heard back from. What is it, you can lead a horse to water, but you can't make him drink?
Next issue, you already know about: Given a detection, the staff will want a diagnosis of the cause, then the root cause, and then the fix. Well, right, given some topology or some such of the detectors, could do some root cause analysis. But, in practice, diagnosis can be difficult. To ease the work of diagnosis, in each detector try to use some variables that, given an alarm, do give a hint about the cause and diagnosis. Or, have several detectors monitoring one server and, considering them jointly, that is, which ones just gave an alarm and which ones didn't, get some hints on cause -- right, could do more useful research here (I mentioned that).
The only VC that called me back wanted not just my all nicely automated detection but also nicely clear diagnosis and, no doubt, correction. Maybe he also wanted me to give Godzilla a bath, manicure, and rub down, too -- no problem, guys! Godzilla bath coming right up with release 2.0 and the Gold Enterprise Edition! I told the VC that anyone who promised to do a good job automating diagnosis in a real and large server farm or network was, uh, exaggerating what they could do and to stay far away.
Typically diagnosis takes a lot of information about the system being monitored. To do really well at diagnosis, likely need already to have seen all the causes and their symptoms. While we should collect data and make progress where we can, that is, for the more common problems, in general we can't do diagnosis well easily.
Temporarily stick your e-mail address in your HN profile and I will send you the PDF. Also stick in a random string, and I will add it to my profile thus confirming my identity.
Thanks - I considered that but I don't like to put barriers in people's way - especially for free content. I don't like to be too marketing-esque. Just not in my nature. :) A couple of thousand folks signed up to the mailing list using the current approach, which I'm pretty happy with.
1. I decided not to do this because my experience is that people like to do something practical first. I've had a huge response to that chapter - lots of folks have gotten into Riemann that had previously been stuck. That alone is a solid +++ for me.
2. Each chapter contains some discussion of capacity planning for specific tools, where relevant.
3. The capstone chapters (11-13) discusses this, as do the chapters covering logging and application instrumentation.
4. Thanks - I'll consider that.
5. I discuss in various chapters visualization but I've found that most folks have very different needs and desires. So I focussed on discussing what to show in small segments as well as some visual design discussion rather than a specific chapter on dashboarding/reporting. Hard choice but a 750 page book needs to stop somewhere. :)
Thanks for taking the time to comment - it's awesome when folks share their thoughts!
That's mostly correct. I have a chapter on adding instrumentation to applications with examples in Ruby/Rails and Clojure that can easily be adapted to other languages and frameworks. I also cover adding structured logging to your applications.
