ludo-technologies/pyscn

Track: tuning · detector: clone Summary Groups created by appendUncoveredPairGroups (#525) are never checked by the covered-group dedup (#518). So the nested-window duplicates that #518 removed come back as separate groups and pairs, and they inflate duplicated_fragments and code_duplication_percentage. Evidence pyscn v1.32.3 (b3a74d7), lordmauve/wasabi2d@78116b4. effects/posterize.py:21-31 (method draw) and posterize.py:23-31 (the with block inside it) are separate fragments. redundant group sim covered by {base_matrix.py:32-41, posterize.py:23-31} 0.768 {base_matrix 32-41, outline 24-33, posterize 21-31} 0.84 {outline.py:24-33, posterize.py:23-31} 0.676 same {punch.py:25-33, posterize.py:23-31} 0.683 {posterize 21-31, punch 25-33} 0.782 (itself appended) That makes 3 of 12 groups and 3 of 19 pairs repeats. duplicated_fragments is 20 instead of 19, so dup % is 4.535 instead of 4.308. Minimal repro Copy wasabi2d/effects/{base_matrix,outline,posterize,punch}.py into an em

Problem extractMethodCalls (internal/analyzer/lcom.go) only sees explicit self.name(...) calls. Methods that reach other methods through Python protocols get no edge: list(self) / for k in self -> __iter__, self[k] -> __getitem__/__setitem__, self == other -> __eq__. Mapping and sequence classes split into many single-method components. Repro class OrderedDictLike: def __init__(self): self.__map = {} def __setitem__(self, key, value): self.__map[key] = value def __iter__(self): return iter(self.__map) def __getitem__(self, key): return self.__map[key] def keys(self): return list(self) def values(self): return [self[key] for key in self] def __eq__(self, other): return dict(self) == dict(other) def __ne__(self, other): return not self == other def update(self, other): for key in other: self[key] = other[key] pyscn analyze --json --no-open --select lcom . Expect

Problem LCOM4 joins methods through shared self.x access or self.method() calls. A method that reads no instance variable and calls no sibling method always becomes its own component. This is normal for operator dunders that pass self to a free function (__add__ -> chain(self, other)), for closure-based wrappers (__call__/__repr__/__str__ using captured locals), and for stdlib overrides like tzinfo.utcoffset/dst/tzname. The class gets a high LCOM4 with nothing to split. Repro def make_operator(name, module): class OperatorImplementation: def __call__(self, *args): return f"{module}.{name}(...)" def __repr__(self): return f"{module}.{name}" def __str__(self): return f"{name}" return OperatorImplementation() pyscn analyze --json --no-open --select lcom . Expected LCOM4 1, or stateless methods left out of the graph the same way abstract methods are (#559). Actual {"name":"OperatorImplementation","lcom4":3,"total_method

Summary The current Type-2 clone detection uses Jaccard coefficient, which treats all features equally weighted. By switching to TF-IDF + Cosine Similarity, we can reduce the weight of frequently occurring tokens and potentially reduce false positives. Current Issues Problems with Jaccard Coefficient All features are treated equally Frequently occurring tokens like @dataclass, def __init__, return carry high weight Computed with the same weight as unique business logic Boilerplate reduction not applied to Type-2 IsBoilerplateLabel() in framework_patterns.go is only used for Type-3 (APTED) cost calculation Not applied in Type-2 Jaccard calculation Causes of false positives Classes using framework patterns (dataclass, Pydantic, etc.) are easily judged as similar Example: Domain-specific logic like ConsumptionTaxPolicy may be falsely detected as a clone of another class with the same decorators Proposal: TF-IDF + Cosine Similarity Benefits of TF-IDF Token Current (Jaccard