使用SciKit在python中记录文档聚类

My data has huge summary descriptions,which end up becoming 10000’s of words when I apply TF/IDF. Is there any proper way to handle this high dimensional data.

我的第一个建议是,除非你因为内存或执行时间问题而绝对必须这样做.

如果你必须处理它,你应该使用降维(例如PCA)或feature selection(在你的情况下可能更好,例如参见chi2)

K – means and other algorithms requires I specify the no. of clusters ( centroids ),in my case I do not know the number of clusters upfront. This I believe is a completely unsupervised learning. Are there algorithms which can determine the no. of clusters themselves?

如果你看一下the clustering algorithms available in scikit-learn,你会发现并非所有这些都要求你指定集群的数量.

另一个不是层次聚类,implemented in scipy.另见this answer.

我还建议您使用KMeans并尝试手动调整群集数量,直到您对结果满意为止.

I’ve never worked with document clustering before,if you are aware of tutorials,textbooks or articles which address this issue,please feel free to suggest.

Scikit有许多处理文本数据的教程,只需在他们的网站上使用“文本数据”搜??索查询.一个用于KMeans,另一个用于监督学习,但我建议你仔细阅读这些以便更熟悉图书馆.在我看来,从编码,风格和语法POV来看,无监督和监督学习在scikit-learn中非常相似.

Document clustering is typically done using TF/IDF. Which essentially converts the words in the documents to vector space model which is then input to the algorithm.

这里的小修正：TF-IDF与聚类无关.它只是一种将文本数据转换为数字数据的方法.之后,它并不关心您对该数据(聚类,分类,回归,搜索引擎等)的处理方式.

我理解你试图传达的信息,但说“使用TF-IDF完成聚类”是不正确的.它使用聚类算法完成,TF-IDF仅在文档聚类中起预处理作用.