shop

Adding products to a cart and filling checkout forms on a live store

Reading the page in the browser

SHOP_JS is a script that the module injects into the store’s page. It finds these items:

  • the product cards, and the add-to-cart button and quantity control on each card
  • the cart count and subtotal, and the cart lines
  • the form fields
  • anything in the way, such as a cookie banner

The script reads the page’s links, text and common markup. It has no rules written for a single store. A page navigation removes the script. The Python code injects it again when it is needed.

Site hints

SITES holds facts that the page does not show: the search and cart URLs of some stores, and notes such as “pick a store first”. site_hint returns the entry for a URL’s host, or {} for a store that is not listed. Unlisted stores also work. The module searches them through the store’s own search box.

source

site_hint

def site_hint(
    url:str
)->dict:

Return the search URL, cart URL and notes known for the host of url, or {} when the store is not listed.

Reading the store

These methods are added to fastcdp’s Page. Each returns plain data.

shop_prep injects SHOP_JS when the page has no copy of it. It returns the platform the store runs on: shopify, woocommerce, bigcommerce, magento, next or generic.

shop_products lists the products on the page. Each product has a number in i. You choose a product to add with that number or with its title. On a product’s own page the main product comes first. The other products on that page are marked related.

shop_cart reads the cart’s item count and subtotal. It uses Shopify’s /cart.js when the store has one and the page otherwise. The source key says which. shop_lines reads the lines on the cart page.

shop_blockers names what is in the way: cookie-banner, location-required, login-required, captcha or modal-open. shop_dismiss clicks through cookie banners and closable pop-ups.

A request that cannot be carried out raises ShopError. An example is a product number that is not on the page. The error lists the choices that were there.

source

Page.shop_dismiss

async def shop_dismiss(
    page:fastcdp.core.Page
)->list:

Click through cookie banners and closable pop-ups and return the labels clicked.

source

Page.shop_blockers

async def shop_blockers(
    page:fastcdp.core.Page
)->list:

Return the names of anything blocking the store, such as a cookie banner or a login requirement.

source

Page.shop_lines

async def shop_lines(
    page:fastcdp.core.Page
)->list:

Return the lines of the cart as shown on the cart page.

source

Page.shop_cart

async def shop_cart(
    page:fastcdp.core.Page, lines:bool=False
)->dict:

Return the cart’s item count and subtotal, and its lines when asked, from Shopify’s /cart.js where available.

source

Page.shop_products

async def shop_products(
    page:fastcdp.core.Page, limit:int=40
)->list:

Return the products on the current page as numbered dictionaries.

source

Page.shop_prep

async def shop_prep(
    page:fastcdp.core.Page, force:bool=False
)->str:

Inject the in-page helpers if the page has none and return the store’s platform.

source

ShopError

def ShopError(
    *args, **kwargs
):

A shop action could not be carried out, such as a product number that is not on the page.

Actions that check their result

shop_add adds a product, chosen by number or by title. It returns ok, how, the product, and the cart before and after.

ok Meaning
True A cart signal changed. A cart signal is the item count, the subtotal, the cart lines or a new confirmation message. how names it: count, subtotal, lines, or the confirmation message.
False Nothing changed. error gives the reason.
None The page shows no cart count or subtotal to compare. The add could not be confirmed.

A title must match a product title exactly, or else it must be part of a title. The exact match wins. A Shopify product can have several variants, such as sizes. When several variants are in stock and you pass no variant, shop_add returns need='variant' and the list of variants in stock. It does not pick one. A product with one variant in stock uses that variant.

shop_cart_page opens the cart page and returns the cart with its lines. shop_qty sets the quantity of a line. shop_remove removes a line. Both check the cart in the same way as shop_add.

source

Page.shop_add

async def shop_add(
    page:fastcdp.core.Page, item, qty:int=1, variant:str=None, open_product:bool=True, tout:int=12
)->dict:

Add a product, chosen by number or title, to the cart and check that the cart changed.

source

Page.shop_remove

async def shop_remove(
    page:fastcdp.core.Page, line, tout:int=12
)->dict:

Remove a cart line and check that the cart changed.

source

Page.shop_qty

async def shop_qty(
    page:fastcdp.core.Page, line, qty:int, tout:int=12
)->dict:

Set the quantity of a cart line and check that the cart changed.

source

Page.shop_cart_page

async def shop_cart_page(
    page:fastcdp.core.Page, url:str=None
)->dict:

Open the cart page and return the cart with its lines.

Checkout forms

shop_fields lists every visible form field on the page. Each entry has the field number i, the label, the autocomplete token and the current value. A drop-down entry also lists its options. shop_fields never returns password values.

shop_fill fills a form from a profile dictionary. The standard keys are first_name, last_name, email, phone, address1, address2, city, state, postcode, country, company, notes and the card_* keys. match_fields decides which field each key goes into. It checks the field’s autocomplete token first, then its name or id, then its label. It uses the words in FIELD_MAP, which include Australian wording such as “suburb” and “postcode”. Each field is used for one key at most.

A numeric key, such as '7', fills field number 7 from shop_fields. Use it for choices that have no standard key, such as size or delivery time.

After the fill, shop_fill reads the form again and reports whether each field holds the value. shop_set sets one field by its number.

shop_fill(submit=...) clicks a button or link by its name after the fill. A name that matches PAY_RX, such as “Pay now” or “Place order”, needs confirm=True. Without it, shop_fill raises ShopError and fills nothing.

source

match_fields

def match_fields(
    fields:list, profile:dict
)->dict:

Return the best form field for each profile key as {key: field}, using each field at most once.

source

Page.shop_fill

async def shop_fill(
    page:fastcdp.core.Page, profile:dict, submit:str=None, confirm:bool=False
)->dict:

Fill a checkout form from a profile dictionary, read it back to confirm each field, and optionally click a button.

source

Page.shop_set

async def shop_set(
    page:fastcdp.core.Page, i:int, value
)->dict:

Set form field i to value and return what the field holds afterwards.

source

Page.shop_fields

async def shop_fields(
    page:fastcdp.core.Page
)->list:

Return every visible form field with its number, label, autocomplete token, value and drop-down options.

Shop

The methods above are async and need a Page. Shop wraps them as plain synchronous methods with short names: platform, products, cart, cart_page, lines, add, set_qty, remove, fields, set, fill, blockers, dismiss, snapshot and md. Each takes simple arguments. The return type depends on the method. platform, snapshot and md return strings. The other listed methods return dictionaries or lists.

Shop also has goto, open, find and search. search tries the store’s known search URL, then Shopify’s search page, then the search box on the page, in that order.

shop(url) opens a session. It connects to the debug Chrome and starts it when needed. It reuses a tab that an earlier shop call left open on the same store. With resume=True it leaves that tab on its current page instead of going to url. The fossick shop --add 0 command uses this to act on the results that a previous --search command left on screen.

source

shop

def shop(
    url:str=None, port:int=9223, headless:bool=None, extra_flags:NoneType=None, resume:bool=False, tout:int=180
)->__main__.Shop:

Open a Shop session on the debug Chrome, reusing a tab already open on the store.

source

Shop

def Shop(
    page, port:int=9223, tout:int=180
):

A synchronous shopping session on the debug Chrome for searching, adding to the cart and checking out.

source

Tests

The offline tests cover the site hints, matching profile keys to a realistic checkout form, the rule for when a cart has changed, choosing a product by number or title, the payment button guard, and cart line lookup. The cells under Live stores need the debug Chrome and the network and are skipped by the tests.

assert site_hint('https://www.coles.com.au/search/products?q=milk')['cart'].endswith('/trolley')
assert site_hint('https://www.amazon.com.au/s?k=x')['search'] == 'https://www.amazon.com.au/s?k={q}'
assert site_hint('https://members.ceresfairfood.org.au/products/search')['note']
assert site_hint('https://some-random-shopify-store.com') == {}     # generic path, no hints needed
#: a checkout form in the shape real ones come in: AU wording, mixed autocomplete coverage, decoys
FORM = [
    dict(i=0, tag='input', type='search', name='q', id='', autocomplete=None, label='Search parts', options=None),
    dict(i=1, tag='input', type='email', name='customer[email]', id='e', autocomplete='email', label='Email address', options=None),
    dict(i=2, tag='input', type='text', name='firstName', id='', autocomplete='off', label='', options=None),
    dict(i=3, tag='input', type='text', name='ship[ln]', id='', autocomplete='family-name', label='Family name', options=None),
    dict(i=4, tag='input', type='tel', name='contact', id='', autocomplete='tel', label='Email or phone', options=None),
    dict(i=5, tag='input', type='text', name='address1', id='', autocomplete=None, label='Address', options=None),
    dict(i=6, tag='input', type='text', name='address2', id='', autocomplete=None, label='Address line 2', options=None),
    dict(i=7, tag='input', type='text', name='locality', id='', autocomplete='address-level2', label='Suburb', options=None),
    dict(i=8, tag='select', type='select-one', name='region', id='', autocomplete='address-level1', label='State / Territory',
         options=[dict(v='VIC', t='Victoria'), dict(v='NSW', t='New South Wales')]),
    dict(i=9, tag='input', type='text', name='zip', id='', autocomplete='postal-code', label='Postcode', options=None),
]
hit = match_fields(FORM, dict(email='a@b.co', first_name='Sam', last_name='Nguyen', phone='0400 000 111',
                              address1='12 Smith St', address2='Unit 3', city='Brunswick',
                              state='Victoria', postcode='3056'))
assert {k: v['i'] for k, v in hit.items()} == {'email': 1, 'first_name': 2, 'last_name': 3, 'phone': 4,
                                              'address1': 5, 'address2': 6, 'city': 7, 'state': 8, 'postcode': 9}
assert hit['city']['label'] == 'Suburb'          # not "City", and found by autocomplete token
assert hit['first_name']['name'] == 'firstName'  # autocomplete="off" -> fall back to the name

# a field's own autocomplete token wins over a misleading label: i=4 says tel, so email skips it
assert _score(FORM[4], 'email') == 0 and _score(FORM[4], 'phone') == 100
assert _score(FORM[0], 'first_name') == 0        # the search box matches nothing
# one field is never claimed twice, and unmatched keys are reported rather than silently dropped
assert len({v['i'] for v in hit.values()}) == len(hit)
assert match_fields(FORM, dict(card_cvc='123')) == {}
#: an explicit variant beats the card's variant id, and an unmatched one is rejected
class _VPage:
    async def eval(self, js): return 'https://x/p' if js == 'location.href' else [dict(id='1', name='Small'), dict(id='2', name='Large')]
_card = dict(url='https://x/p', vid='1')
assert (await _variant_id(_VPage(), _card, 'Large'))[0] == '2'
assert (await _variant_id(_VPage(), _card, 'Huge'))[0] is None
assert (await _variant_id(_VPage(), _card, None))[0] == '1'
assert (await _variant_id(_VPage(), _card, 'L'))[0] is None   # both names contain 'L' and neither equals it: no guess
assert (await _variant_id(_VPage(), _card, 'large'))[0] == '2'

class _XLPage:
    async def eval(self, js): return 'https://x/p' if js == 'location.href' else [dict(id='3', name='X-Large'), dict(id='4', name='Large')]
assert (await _variant_id(_XLPage(), _card, 'Large'))[0] == '4'   # an exact name wins over one that merely contains it
assert (await _variant_id(_XLPage(), _card, 'X-L'))[0] == '3'

class _NoVariants:
    async def shop_prep(self): return 'shopify'
    async def shop_products(self): return [dict(i=0, title='Tee', url='https://x/p', vid='1', add=True)]
    async def eval(self, js): return 'https://x/p' if js == 'location.href' else []
res = await Page.shop_add(_NoVariants(), 0, 1, 'Large')
assert res['ok'] is False and res['need'] == 'variant' and 'nothing was added' in res['error']
#: verification: only a real change counts, and only in a signal that the page exposes
assert _changed(dict(count=0, subtotal=0), dict(count=1, subtotal=9.5)) == 'count'
assert _changed(dict(count=None, subtotal=10.0), dict(count=None, subtotal=19.5)) == 'subtotal'
assert _changed(dict(count=1, subtotal=9.5), dict(count=1, subtotal=9.5)) == ''
assert _changed(dict(count=1), dict(count=None)) == ''     # a reading that went blank is not progress
assert _changed(dict(count=1, lines=[{'title': 'a', 'qty': 1}]),
                dict(count=1, lines=[{'title': 'a', 'qty': 1}, {'title': 'b', 'qty': 1}])) == 'lines'
#: picking a product never guesses. An unusable request raises and lists what was on offer.
ITEMS = [dict(i=0, title='Oil Filter Z145A', related=None), dict(i=1, title='Oil Filter Z411', related=None),
         dict(i=2, title='Oil Filter Z411 Twin Pack', related=None), dict(i=3, title='Oil Filter Z145A', related=True)]
assert _match(ITEMS, 1)['i'] == 1
assert _match(ITEMS, '1')['i'] == 1                        # a stringified index
assert _match(ITEMS, 'Oil Filter Z411')['i'] == 1          # exact title beats the longer superstring
assert _match(ITEMS, 'twin pack')['i'] == 2                # substring, case-insensitive
assert _match(ITEMS, 'Oil Filter Z145A')['i'] == 0         # the real product, not the recommendation
for bad in ('flux capacitor', 9):
    try:
        _match(ITEMS, bad)
        raise AssertionError(f'{bad!r} should not have matched')
    except ShopError as e: assert 'Z145A' in str(e) or 'page has 4' in str(e), e
#: the payment guard covers the wording checkouts actually use
for label in ('Pay now', 'Place order', 'Complete order', 'Confirm purchase', 'Buy now', 'Submit order'):
    assert PAY_RX.search(label), label
for label in ('Continue to shipping', 'Save address', 'Apply discount', 'Update cart'):
    assert not PAY_RX.search(label), label
#: line lookup and numeric fill keys. Both name what is on the page and never guess.
CART = dict(count=2, subtotal=13.0, lines=[dict(i=0, text='Apples', qty='2', qty_kind='select', remove=True),
                                           dict(i=1, text='Pears', qty='1', qty_kind=None, remove=True)])
assert _line(CART, 1)['text'] == 'Pears' and _line(CART, '0')['qty_kind'] == 'select'
try:
    _line(CART, 5)
    raise AssertionError('line 5 should not have resolved')
except ShopError as e: assert 'cart page shows 2' in str(e), e

#: a numeric profile key addresses a `fields()` index and never goes through `FIELD_MAP`
assert match_fields(FORM, {'7': 'Large', 'city': 'Brunswick'})['city']['i'] == 7   # '7' is not a profile key
assert _origin('https://shop.example.com/collections/all?x=1') == 'https://shop.example.com'

Live stores

These run against Allbirds, a Shopify store, and CERES Fair Food, a custom store, and are skipped by the tests.

s = shop('https://www.allbirds.com')
s.search('tree runner')[:2]
# no size chosen and several in stock: it asks instead of picking one
r = s.add(0)
r.get('need'), [v['name'] for v in (r.get('variants') or [])][:3]
s.add(0, qty=2, variant='9')        # -> {'ok': True, 'how': 'count', 'variant_id': '331912...', ...}
s.cart(lines=True), s.set_qty(0, 1), s.remove(0)
# CERES Fair Food: custom Rails storefront, "Add" buttons, quantity selects
c = shop('https://members.ceresfairfood.org.au')
[(p['title'], p['price'], p['qty']) for p in c.search('apples')[:3]]
# logged out there is no cart signal to check, so the result says so rather than claiming success
c.add('Apples Fuji Organic 500g', qty=2)   # -> {'ok': None, 'error': 'no cart signal ...', 'blockers': [...]}

Limits

fossick does not solve captchas, and it never clicks a payment button unless the call passes confirm=True. Stores change their pages over time and the in-page script may stop finding a store’s controls, which is why every action checks the cart and reports what it saw rather than assuming the click worked.