chore: import upstream snapshot with attribution
Tests / tests (map[TOXENV:py310], macos-latest, 3.10) (push) Has been cancelled
Tests / tests (map[TOXENV:py311], macos-latest, 3.11) (push) Has been cancelled
Tests / tests (map[TOXENV:py312], macos-latest, 3.12) (push) Has been cancelled
Tests / tests (map[TOXENV:py313], macos-latest, 3.13) (push) Has been cancelled
Tests / tests (map[TOXENV:py310], macos-latest, 3.10) (push) Has been cancelled
Tests / tests (map[TOXENV:py311], macos-latest, 3.11) (push) Has been cancelled
Tests / tests (map[TOXENV:py312], macos-latest, 3.12) (push) Has been cancelled
Tests / tests (map[TOXENV:py313], macos-latest, 3.13) (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,68 @@
|
||||
# Writing your retrieval system
|
||||
|
||||
Scrapling uses SQLite by default, but this tutorial shows how to write your own storage system to store element properties for the `adaptive` feature.
|
||||
|
||||
You might want to use Firebase, for example, and share the database between multiple spiders on different machines. It's a great idea to use an online database like that because spiders can share adaptive data with each other.
|
||||
|
||||
So first, to make your storage class work, it must do the big 3:
|
||||
|
||||
1. Inherit from the abstract class `scrapling.core.storage.StorageSystemMixin` and accept a string argument, which will be the `url` argument to maintain the library logic.
|
||||
2. Use the decorator `functools.lru_cache` on top of the class to follow the Singleton design pattern as other classes.
|
||||
3. Implement methods `save` and `retrieve`, as you see from the type hints:
|
||||
- The method `save` returns nothing and will get two arguments from the library
|
||||
* The first one is of type `lxml.html.HtmlElement`, which is the element itself. It must be converted to a dictionary using the `element_to_dict` function in the submodule `scrapling.core.utils._StorageTools` to maintain the same format, and then saved to your database as you wish.
|
||||
* The second one is a string, the identifier used for retrieval. The combination result of this identifier and the `url` argument from initialization must be unique for each row, or the `adaptive` data will be messed up.
|
||||
- The method `retrieve` takes a string, which is the identifier; using it with the `url` passed on initialization, the element's dictionary is retrieved from the database and returned if it exists; otherwise, it returns `None`.
|
||||
|
||||
> If the instructions weren't clear enough for you, you can check my implementation using SQLite3 in [storage_adaptors](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/storage.py) file
|
||||
|
||||
If your class meets these criteria, the rest is straightforward. If you plan to use the library in a threaded application, ensure your class supports it. The default used class is thread-safe.
|
||||
|
||||
Some helper functions are added to the abstract class if you want to use them. It's easier to see it for yourself in the [code](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/storage.py); it's heavily commented :)
|
||||
|
||||
|
||||
## Real-World Example: Redis Storage
|
||||
|
||||
Here's a more practical example generated by AI using Redis:
|
||||
|
||||
```python
|
||||
import redis
|
||||
import orjson
|
||||
from functools import lru_cache
|
||||
from scrapling.core.storage import StorageSystemMixin
|
||||
from scrapling.core.utils import _StorageTools
|
||||
|
||||
@lru_cache(None)
|
||||
class RedisStorage(StorageSystemMixin):
|
||||
def __init__(self, host='localhost', port=6379, db=0, url=None):
|
||||
super().__init__(url)
|
||||
self.redis = redis.Redis(
|
||||
host=host,
|
||||
port=port,
|
||||
db=db,
|
||||
decode_responses=False
|
||||
)
|
||||
|
||||
def save(self, element, identifier: str) -> None:
|
||||
# Convert element to dictionary
|
||||
element_dict = _StorageTools.element_to_dict(element)
|
||||
|
||||
# Create key
|
||||
key = f"scrapling:{self._get_base_url()}:{identifier}"
|
||||
|
||||
# Store as JSON
|
||||
self.redis.set(
|
||||
key,
|
||||
orjson.dumps(element_dict)
|
||||
)
|
||||
|
||||
def retrieve(self, identifier: str) -> dict | None:
|
||||
# Get data
|
||||
key = f"scrapling:{self._get_base_url()}:{identifier}"
|
||||
data = self.redis.get(key)
|
||||
|
||||
# Parse JSON if exists
|
||||
if data:
|
||||
return orjson.loads(data)
|
||||
return None
|
||||
```
|
||||
@@ -0,0 +1,22 @@
|
||||
# Using Scrapling's custom types
|
||||
|
||||
> You can take advantage of the custom-made types for Scrapling and use them outside the library if you want. It's better than copying their code, after all :)
|
||||
|
||||
### All current types can be imported alone, like below
|
||||
```python
|
||||
from scrapling.core.custom_types import TextHandler, AttributesHandler
|
||||
|
||||
somestring = TextHandler('{}')
|
||||
somestring.json() # '{}'
|
||||
somedict_1 = AttributesHandler({'a': 1})
|
||||
somedict_2 = AttributesHandler(a=1)
|
||||
```
|
||||
|
||||
Note that `TextHandler` is a subclass of Python's `str`, so all standard operations/methods that work with Python strings will work.
|
||||
If you want to check the type in your code, it's better to use Python's built-in `issubclass` function.
|
||||
|
||||
The class `AttributesHandler` is a subclass of `collections.abc.Mapping`, so it's immutable (read-only), and all operations are inherited from it. The data passed can be accessed later through the `_data` property, but be careful; it's of type `types.MappingProxyType`, so it's immutable (read-only) as well (faster than `collections.abc.Mapping` by fractions of seconds).
|
||||
|
||||
So, to make it simple for you, if you are new to Python, the same operations and methods from the Python standard `dict` type will all work with the class `AttributesHandler` except for the ones that try to modify the actual data.
|
||||
|
||||
If you want to modify the data inside `AttributesHandler`, you have to convert it to a dictionary first, e.g., using the `dict` function, and then change it outside.
|
||||
Reference in New Issue
Block a user