I built Slopo to solve one specific problem: finding similar code that is hardest to detect by other tools, coding AI agents, and humans.
It finds similar-looking code with embeddings. This detects more than just copy-paste clones or even clones with minor changes. Similar code is often not a clone to refactor, and this is a trade-off. Initial results need to be verified, but coding agents can do this quickly. Example prompts are available on https://slopo.dev
Additionally, similar code distant in the codebase is ranked higher to focus on less obvious duplication.
The results differ a lot depending on the codebase. I noticed that sometimes most of the detected duplicates are false positives, but the remaining ones are strong candidates to refactor or even bugs. Sometimes it reveals much more real duplication.
Correct me if I'm wrong, but looking at [1] it seems to be specifically using function definitions (I'm guessing this works with functions, methods, and lambdas (the "<unknown>" part)?) as units of repetition. If yes, that's fine, but I would seriously consider adding some settings to allow the user to control that granularity. Sometimes, the repeated code is a conditional branch within larger functions (i.e., "every else:" or "every except Ex:" looks the same). If the functions are large enough, the dissimilarity of the rest of the body would (probably?) cause such things to be missed.
I would also consider - perhaps as a separate pass, with scoring set differently - to analyze comments (especially docstrings in Python). If I read the code correctly, you're currently just stripping them, which is the right thing to do when looking for code duplication, but duplicated docstrings are also often a signal that something is wrong in the codebase. The "different scoring" is because we expect docstring to be structured similarly (at least more than normal code), so some tweaking would be needed.
Currently, only whole functions (including function-like constructs depending on language) are considered as unit.
Skipping the extraction of conditional branches was my decision to not overcomplicate the first versions, which was intended to validate the idea. I will add this in future versions because I agree it's needed for large functions.
I don't think it needs configurable granularity. In the current version, there is an analogous mechanism: when functions are nested, both outer and inner are embedded separately. When both are similar to each other, this pair is excluded. Inner or outer functions can appear in results depending on similarity to other units.
Regarding comments, they are removed and I will think about handling them. The challenge is not with extraction, but with how to present this in a report. This may be a nice addition because coding agents often add comments.
The false positive rate you're describing matches what we see running similarity detection on generated text instead of code: cosine similarity alone flags a lot of same-topic pairs that aren't actually duplicates. What helped was combining the embedding score with a structural signal (AST edit distance for code, overlapping headings and citations for text) so no single metric makes the call. Also worth surfacing the raw similarity score in the CLI output instead of just a binary duplicate flag, since people will want to tune the threshold per codebase.
1. The tool uses only cosine similarity plus boost depending on distance in the codebase.
2. Classification with LLM. This can be done by coding agent used with project giving better results than integrating this pass in the tool. LLMs used for coding are pretty good.
I assumed that this is not a problem I need to solve inside the tool. I'm aware this is not deterministic, but this is by design.
Regarding information about raw similarity: currently, the score (raw similarity + boost) is visible in the report, so this value can be configured based on data. The raw similarity threshold can also be configured, but it's not displayed. I will think about how to handle this.
Cool project, I've been meaning to do this myself at work for a codebase, and it's nice to see that this exists now.
Does the project you simply compute embeddings for every function unit and cluster them, or do we also mean-pool significant dependencies of a function? In other words, given the function
def a():
b()
c()
d()
Do we also embed b, c, and d as well and combine them somehow in the embedding of a?
It looks like it works only on function bodies[1]. I'm not sure I understand why you would want to look at invoked callables code, though. Calling the same set of helper functions is already flagged; repeated code in helpers is flagged as well when those helpers are analyzed. Do you have a specific example where you'd like a function flagged as a duplicate based on the code it calls out to?
Based on your example there is only a single function a() which is embedded. The rest is just a code and dependencies are not resolved. Did you think about adding this feature in your tool?
For false positives, how about generating the test units and run it in isolation?
All major languages have the interpreters or embedded languages to execute function only.
Do you mean to verify if similar code units produce the same result?
The goal of the tool is to also detect code that is similar and behaves differently. There are not ideal duplicates, but still a code that can be refactored, abstracted, or fixed (because the variance may be the result of bug).
I implemented this for a large monorepo last year, it runs as an analysis during code review and it shows what are possible similar snippets wrt the code under review. It was a very nice project. It also allows to see across the repo what are the most common constructs for the different languages. This could also be helpful to see if some code has been copied e.g. from open source projects.
Finding similar code is something different than deduplication, even when the final goal looks similar.
Deduplication backend is the easiest part of the tool and it doesn't need any additional libraries. Just calculate embeddings and find close pairs. The complexity is everything around.
Using local models is worth considering and the tool already uses the LiteLLM wrapper, allowing it to configure different models, including local. I left this part for the user.
I don't understand what you mean with your first sentence. Both SemHash and Slopo are deduplication libraries right? Regardless, finding similar code is the core functionality that enables (semantic) deduplication.
I also don't think the backend is the "easiest part" here, a lot of the scalability lives there, which is important for monorepos, or cases where you want to deduplicate across projects. For example, from looking at the implementation, you use exact brute-force similarity search (comparing every item to every other item) which is an O(n^2) operation. It also allocates large dense similarity blocks in memory, so memory use won’t scale well either.
Slopo doesn't deduplicate. It only reports similar code units, which in most cases are not duplicates. This needs to be cleaned up by the user, which is fast and easy with coding agents.
Embedding calculation is also outsourced externally: it's only a simple API call with a LiteLLM wrapper.
"brute-force similarity search" and O(n^2) may sound scary, but it works fine and this is not a bottleneck. For large projects, other parts are much slower, which has room for improvement. [1] is an implementation you probably saw. It uses NumPy, spreads work across all CPU cores and there is also a split into blocks (block_size = 1000). In larger sets, all vectors are not loaded into memory at once. Where I need to be honest, I didn't measure actual memory usage. I just tested this on large repos, so I'm aware of bottlenecks.
From my perspective, this is the easiest part. Code extraction, chunking, applying boost, clustering, generating report and designing everything as a single user-friendly tool is a real challenge. Architectural and product decisions are more difficult than implementation and solving performance issues.
"it works fine and this is not a bottleneck. For large projects, other parts are much slower": I don't think this is true, especially for large projects. I just ran your tool on the Kubernetes repo with 1536-dim embeddings. The isolated similar-pair search took ~130s and peaked at ~4.8 GB RSS, and the total runtime was ~250s with the same memory peak.
"In larger sets, all vectors are not loaded into memory at once": this is also not correct, at least in the implementation you shared. The similarity matrix is processed in blocks, but the embeddings themselves are loaded all at once and stacked into one NumPy matrix, hence the memory peak.
So in larger projects, more than half of the time is spent on the similarity search step, and almost all the memory is spent there as well.
To give more context, the current version is already an optimized one I considered good enough and didn't spend more time on it. In the first attempt, I used a vector database with indexes, trying to query for similar vectors. This was uselessly slow even in medium-sized repos. The brute-force NumPy solution is a significant improvement, making it faster than other calculations like clustering.
"vectors are not loaded into memory at once" is not true, I had in mind splitting computation into blocks.
One possible simple optimization is to use 16-bit floats in vector instead of 32-bit. I used this in a different project (halfvec in pgvector) without affecting results.
1536-dim embeddings from your case also can be reduced. This large vector usually doesn't give much benefit compared to smaller ones. And this is something I will compare in my own tests.
I've used jscpd, it does "non-exact code duplication" (regardless of what its readme says) which is why I asked how Slopo compares. I'm surprised you've not tried the competition!
* cluster-01.md has the highest similarity, and jscpd probably will detect this too.
* cluster-10.md has the lowest similarity still above the threshold, and I don't think jscpd will detect this as clones. Because they are not clones, this is a false positive that needs to be discarded. But in other cases this kind of similar code may be worth acting on.
I didn't compare with jscpd because I don't consider it a competition. Embedding-based duplication detection works differently, gives different results and has its own trade-offs.
No. It depends the most on general code quality and architecture. Some implementations require more code similarity by design. Some languages, like Java, may tend to have more duplication, but it's only a theoretical guess. It also depends on what kind of software is developed with what language.
I just focused on embeddings without comparing them to deterministic solutions.
But I plan to do my own analysis of different embedding models in the context of code similarity detection. Including BM25 in the comparison is a very good idea.
There are good mature tools for deterministic duplication detection and I intentionally focused on embedding-based to fill this gap (I didn't find other tools using this approach).
If by "more efficient" you mean to avoid embedding of the same code multiple times, this optimization is already implemented internally.
This is a great use case for embeddings. Code deduplication across distant modules is notoriously hard for traditional AST-based tools.
How do you handle chunking and parsing for different languages to make sure the embeddings capture semantic meaning effectively? For instance, do you chunk by functions/classes, or use a fixed token window? If a function is too long or too short, it can drastically skew the embedding similarity.
Generally, I chunk by function/method (not by whole class), but different languages have specific concepts and features. Nested code units, anonymous functions, lambdas, closures are extracted as separate chunks.
The chunk size has allowed range and those outside are simply ignored.
- Upper limit is hardcoded with a body size of 10k chars
- Lower limit is configurable with a default of 10 AST nodes inside the body
The chunking strategy is something that can be improved in future versions.
Recently there was a popular article on HN saying that sometimes code duplication is better than abstraction, so I assume that this question is not a joke.
While testing this tool, one detected duplication was interesting for a use case. Permission check logic was duplicated and placed in different distant places in the codebase. The code was similar, but not identical, the logic was not the same. One version had stricter checks. I analyzed this with the coding agent, and we found out that both versions are used for the same thing, which means that in some cases validation is insufficient. Having only a single validation place, this bug could be prevented or easily detected.
I seem to have misunderstood. I thought that it was talking about stuff like similar repos, but now I realize that this is most likely talking about a singular codebase
I tend to follow the "rule of 3": a second similar implementation is OK, introducing the third triggers a refactor. As with everything, this isn't dogma, and sometimes the second implementation is already too much, while at other times you get tens of similar code sections (in codegen, repeating patterns with almost no changes is a virtue). But it's a good rule of thumb.
On testability: two implementations can be tested against each other, leading to greater coverage with less test code. It doesn't work that way for 3+ implementations, which is another reason not to have that many.
I seem to have misunderstood. I thought that it was talking about stuff like similar repos, but now I realize that this is most likely talking about a singular codebase
I seem to have misunderstood. I thought that it was talking about stuff like similar repos, but now I realize that this is most likely talking about a singular codebase
I built Slopo to solve one specific problem: finding similar code that is hardest to detect by other tools, coding AI agents, and humans.
It finds similar-looking code with embeddings. This detects more than just copy-paste clones or even clones with minor changes. Similar code is often not a clone to refactor, and this is a trade-off. Initial results need to be verified, but coding agents can do this quickly. Example prompts are available on https://slopo.dev
Additionally, similar code distant in the codebase is ranked higher to focus on less obvious duplication.
The results differ a lot depending on the codebase. I noticed that sometimes most of the detected duplicates are false positives, but the remaining ones are strong candidates to refactor or even bugs. Sometimes it reveals much more real duplication.
Correct me if I'm wrong, but looking at [1] it seems to be specifically using function definitions (I'm guessing this works with functions, methods, and lambdas (the "<unknown>" part)?) as units of repetition. If yes, that's fine, but I would seriously consider adding some settings to allow the user to control that granularity. Sometimes, the repeated code is a conditional branch within larger functions (i.e., "every else:" or "every except Ex:" looks the same). If the functions are large enough, the dissimilarity of the rest of the body would (probably?) cause such things to be missed.
I would also consider - perhaps as a separate pass, with scoring set differently - to analyze comments (especially docstrings in Python). If I read the code correctly, you're currently just stripping them, which is the right thing to do when looking for code duplication, but duplicated docstrings are also often a signal that something is wrong in the codebase. The "different scoring" is because we expect docstring to be structured similarly (at least more than normal code), so some tweaking would be needed.
Finally: very nice project, congrats! :)
[1] https://github.com/rafal-qa/slopo/blob/main/src/slopo/indexi...
Currently, only whole functions (including function-like constructs depending on language) are considered as unit.
Skipping the extraction of conditional branches was my decision to not overcomplicate the first versions, which was intended to validate the idea. I will add this in future versions because I agree it's needed for large functions.
I don't think it needs configurable granularity. In the current version, there is an analogous mechanism: when functions are nested, both outer and inner are embedded separately. When both are similar to each other, this pair is excluded. Inner or outer functions can appear in results depending on similarity to other units.
Regarding comments, they are removed and I will think about handling them. The challenge is not with extraction, but with how to present this in a report. This may be a nice addition because coding agents often add comments.
Thanks for the feedback.
What a clever little tool. This is exactly the kind of pragmatic AI tools I want to see more of: linux-y single purpose tools!
If it did PHP I would love to run it over WordPress. What would it take to add that?
PHP was added in the latest release.
PHP support can be easily added, I will release a new version soon.
Thank you
The false positive rate you're describing matches what we see running similarity detection on generated text instead of code: cosine similarity alone flags a lot of same-topic pairs that aren't actually duplicates. What helped was combining the embedding score with a structural signal (AST edit distance for code, overlapping headings and citations for text) so no single metric makes the call. Also worth surfacing the raw similarity score in the CLI output instead of just a binary duplicate flag, since people will want to tune the threshold per codebase.
My solution for false positives is simpler:
1. The tool uses only cosine similarity plus boost depending on distance in the codebase.
2. Classification with LLM. This can be done by coding agent used with project giving better results than integrating this pass in the tool. LLMs used for coding are pretty good.
I assumed that this is not a problem I need to solve inside the tool. I'm aware this is not deterministic, but this is by design.
Regarding information about raw similarity: currently, the score (raw similarity + boost) is visible in the report, so this value can be configured based on data. The raw similarity threshold can also be configured, but it's not displayed. I will think about how to handle this.
You're replying to an LLM bot.
Cool project, I've been meaning to do this myself at work for a codebase, and it's nice to see that this exists now.
Does the project you simply compute embeddings for every function unit and cluster them, or do we also mean-pool significant dependencies of a function? In other words, given the function
Do we also embed b, c, and d as well and combine them somehow in the embedding of a?It looks like it works only on function bodies[1]. I'm not sure I understand why you would want to look at invoked callables code, though. Calling the same set of helper functions is already flagged; repeated code in helpers is flagged as well when those helpers are analyzed. Do you have a specific example where you'd like a function flagged as a duplicate based on the code it calls out to?
[1] https://github.com/rafal-qa/slopo/blob/main/src/slopo/indexi...
Based on your example there is only a single function a() which is embedded. The rest is just a code and dependencies are not resolved. Did you think about adding this feature in your tool?
If your using the embedding models to detect the code duplication then how you will be over coming the Agents hallucination???
For false positives, how about generating the test units and run it in isolation? All major languages have the interpreters or embedded languages to execute function only.
Do you mean to verify if similar code units produce the same result?
The goal of the tool is to also detect code that is similar and behaves differently. There are not ideal duplicates, but still a code that can be refactored, abstracted, or fixed (because the variance may be the result of bug).
I implemented this for a large monorepo last year, it runs as an analysis during code review and it shows what are possible similar snippets wrt the code under review. It was a very nice project. It also allows to see across the repo what are the most common constructs for the different languages. This could also be helpful to see if some code has been copied e.g. from open source projects.
Have you compared this to https://github.com/MinishLab/semhash (or considered using that for the deduplication backend)?
Finding similar code is something different than deduplication, even when the final goal looks similar.
Deduplication backend is the easiest part of the tool and it doesn't need any additional libraries. Just calculate embeddings and find close pairs. The complexity is everything around.
Using local models is worth considering and the tool already uses the LiteLLM wrapper, allowing it to configure different models, including local. I left this part for the user.
I don't understand what you mean with your first sentence. Both SemHash and Slopo are deduplication libraries right? Regardless, finding similar code is the core functionality that enables (semantic) deduplication.
I also don't think the backend is the "easiest part" here, a lot of the scalability lives there, which is important for monorepos, or cases where you want to deduplicate across projects. For example, from looking at the implementation, you use exact brute-force similarity search (comparing every item to every other item) which is an O(n^2) operation. It also allocates large dense similarity blocks in memory, so memory use won’t scale well either.
Slopo doesn't deduplicate. It only reports similar code units, which in most cases are not duplicates. This needs to be cleaned up by the user, which is fast and easy with coding agents.
Embedding calculation is also outsourced externally: it's only a simple API call with a LiteLLM wrapper.
"brute-force similarity search" and O(n^2) may sound scary, but it works fine and this is not a bottleneck. For large projects, other parts are much slower, which has room for improvement. [1] is an implementation you probably saw. It uses NumPy, spreads work across all CPU cores and there is also a split into blocks (block_size = 1000). In larger sets, all vectors are not loaded into memory at once. Where I need to be honest, I didn't measure actual memory usage. I just tested this on large repos, so I'm aware of bottlenecks.
From my perspective, this is the easiest part. Code extraction, chunking, applying boost, clustering, generating report and designing everything as a single user-friendly tool is a real challenge. Architectural and product decisions are more difficult than implementation and solving performance issues.
[1] https://github.com/rafal-qa/slopo/blob/v0.3.0/src/slopo/anal...
"it works fine and this is not a bottleneck. For large projects, other parts are much slower": I don't think this is true, especially for large projects. I just ran your tool on the Kubernetes repo with 1536-dim embeddings. The isolated similar-pair search took ~130s and peaked at ~4.8 GB RSS, and the total runtime was ~250s with the same memory peak.
"In larger sets, all vectors are not loaded into memory at once": this is also not correct, at least in the implementation you shared. The similarity matrix is processed in blocks, but the embeddings themselves are loaded all at once and stacked into one NumPy matrix, hence the memory peak.
So in larger projects, more than half of the time is spent on the similarity search step, and almost all the memory is spent there as well.
Thanks, I will look at this in more detail.
To give more context, the current version is already an optimized one I considered good enough and didn't spend more time on it. In the first attempt, I used a vector database with indexes, trying to query for similar vectors. This was uselessly slow even in medium-sized repos. The brute-force NumPy solution is a significant improvement, making it faster than other calculations like clustering.
"vectors are not loaded into memory at once" is not true, I had in mind splitting computation into blocks.
One possible simple optimization is to use 16-bit floats in vector instead of 32-bit. I used this in a different project (halfvec in pgvector) without affecting results.
1536-dim embeddings from your case also can be reduced. This large vector usually doesn't give much benefit compared to smaller ones. And this is something I will compare in my own tests.
Nice idea. I can see this being useful before refactors, especially when the duplication is semantic rather than copy paste.
How does it compare to jscpd? https://github.com/kucherenko/jscpd
It's the opposite.
jscpd is advertised as "Copy/paste detector", Slopo is advertised as "non-exact code duplication".
Slopo also detects copy/pasted code, but this is not the main goal and the report focuses more on similar code units.
I've used jscpd, it does "non-exact code duplication" (regardless of what its readme says) which is why I asked how Slopo compares. I'm surprised you've not tried the competition!
"Non-exact" may be interpreted in different ways, so let's look at the example report: https://github.com/rafal-qa/slopo/tree/main/doc/example-repo...
* cluster-01.md has the highest similarity, and jscpd probably will detect this too.
* cluster-10.md has the lowest similarity still above the threshold, and I don't think jscpd will detect this as clones. Because they are not clones, this is a false positive that needs to be discarded. But in other cases this kind of similar code may be worth acting on.
I didn't compare with jscpd because I don't consider it a competition. Embedding-based duplication detection works differently, gives different results and has its own trade-offs.
self plug (for similar tool): https://github.com/forhadahmed/refactor
Looks very cool! I'd be very interested in applying this to my Elixir projects. What does it take to add proper support for a new language?
It can be added with https://pypi.org/project/tree-sitter-elixir/ similar to other languages. I will add this and plan to release a new version today.
Elixir was added in the latest release.
This is neat. Have you noticed any difference in duplicate detection between strongly typed and loosely typed languages / code bases?
No. It depends the most on general code quality and architecture. Some implementations require more code similarity by design. Some languages, like Java, may tend to have more duplication, but it's only a theoretical guess. It also depends on what kind of software is developed with what language.
If you are interested in data, you can check my article. Analysis was done with this tool, but a previous version where exact-copy duplicates were excluded from analysis. https://rkochanowski.com/article/analysis-code-duplication/
What a simple and smart idea. Wonderful
Very nice. I can imagine putting this into a pre push hook to keep things clean after an initial sweep.
Nice, what's the chunking level? I would want sub function, logical blocks, etc
Function level only. I will add more granular chunking in next versions.
Did you benchmark it against simpler methods like BM25?
I just focused on embeddings without comparing them to deterministic solutions.
But I plan to do my own analysis of different embedding models in the context of code similarity detection. Including BM25 in the comparison is a very good idea.
have you considered a deterministic tier before the embedding pass? I feel that approach can be more efficient.
We did this by using the ASTs you can go quite far without embeddings and the result is easier to debug and follow what's going on.
There are good mature tools for deterministic duplication detection and I intentionally focused on embedding-based to fill this gap (I didn't find other tools using this approach).
If by "more efficient" you mean to avoid embedding of the same code multiple times, this optimization is already implemented internally.
[dead]
looks super useful- thanks for sharing!
[flagged]
This is a great use case for embeddings. Code deduplication across distant modules is notoriously hard for traditional AST-based tools.
How do you handle chunking and parsing for different languages to make sure the embeddings capture semantic meaning effectively? For instance, do you chunk by functions/classes, or use a fixed token window? If a function is too long or too short, it can drastically skew the embedding similarity.
Generally, I chunk by function/method (not by whole class), but different languages have specific concepts and features. Nested code units, anonymous functions, lambdas, closures are extracted as separate chunks.
The chunk size has allowed range and those outside are simply ignored.
- Upper limit is hardcoded with a body size of 10k chars
- Lower limit is configurable with a default of 10 AST nodes inside the body
The chunking strategy is something that can be improved in future versions.
I think that this is pretty cool, but is there any reason why we would want to remove similar/possible duplicate code?
Recently there was a popular article on HN saying that sometimes code duplication is better than abstraction, so I assume that this question is not a joke.
While testing this tool, one detected duplication was interesting for a use case. Permission check logic was duplicated and placed in different distant places in the codebase. The code was similar, but not identical, the logic was not the same. One version had stricter checks. I analyzed this with the coding agent, and we found out that both versions are used for the same thing, which means that in some cases validation is insufficient. Having only a single validation place, this bug could be prevented or easily detected.
I seem to have misunderstood. I thought that it was talking about stuff like similar repos, but now I realize that this is most likely talking about a singular codebase
(without sarcasm) Is this a serious question?
If so - maintainability, testability. This is old software engineering best practice at this point.
You shouldn’t hyper optimize for deduplication, but it’s usually worth considering. Fewer places to fix issues or improve as well.
I tend to follow the "rule of 3": a second similar implementation is OK, introducing the third triggers a refactor. As with everything, this isn't dogma, and sometimes the second implementation is already too much, while at other times you get tens of similar code sections (in codegen, repeating patterns with almost no changes is a virtue). But it's a good rule of thumb.
On testability: two implementations can be tested against each other, leading to greater coverage with less test code. It doesn't work that way for 3+ implementations, which is another reason not to have that many.
I seem to have misunderstood. I thought that it was talking about stuff like similar repos, but now I realize that this is most likely talking about a singular codebase
Have you written software before?
I seem to have misunderstood. I thought that it was talking about stuff like similar repos, but now I realize that this is most likely talking about a singular codebase