Table mining is a popular research field that involves complicated technologies, including information retrieval, data mining, visual and textual understanding and logical reasoning. With the emergence of Large Language Models (LLMs), the field has witnessed considerable advancements, presenting new paradigms for table understanding, extraction, and reasoning. In this survey, we conduct a comprehensive review of the literature on table mining with LLMs. We begin by introducing the fundamental overview of tabular data and possible challenges in LLM-based table mining. Specifically, we explore the challenges unique to this domain, such as heterogeneous table structures, contextual ambiguity, and domain-specific knowledge requirements. Then, we summarize representative tabular tasks in table preparation and mining, categorizing existing methods along dimensions including task scope, model architecture, and application scenarios. Next, we describe advanced LLM-based learning strategies in table mining, including foundation models and training-free methods. As specific issues, we review studies of trustworthy LLM-based table mining and some domain-specific applications. Finally, we discuss prospects and future directions in the field of LLM-based table reasoning, including issues of generalization, interpretability, efficiency, etc. We hope this survey can present valuable resources for researchers and practitioners, paving the way for further exploration in this field. The repository is at: https://github.com/USTCAGI/Awesome-LLM-Table-Mining.