The difference between GPU and chip mining
Publish: 2021-04-24 07:36:48
1. As the name suggests, the mine calculation card is a graphics card used for mining. More strictly speaking, it is a graphics card used for mining with high load for a long time. The graphics card used in mining usually works at full load 24 hours a day for several months. In this way, PCB and electronic components will accelerate aging, affecting the life of components. Not counting the rest time of the graphics card, even if we play the game for 8 hours a day, the life of the mine card is only one third of that of the normal graphics card. It can be said that the general life of the mine card is only a few months.
2. Mining software, from the programming principle, is about large-scale parallel computing
there are many stream processors in GPU of graphics card. For example, the old HD5770 has 800 stream processors, which is equivalent to 800 computing cores. In recent years, the mainstream high-end graphics card has more built-in stream processors and more powerful computing performance. Using GPU for mining operation, the stream processor is used for parallel operation, and the mining program also optimizes the GPU parallel operation, so the operation efficiency is particularly high, and the efficiency of bitcoin mining has been greatly improved
the number of CPU cores is relatively small. At present, the number of the most powerful mainstream desktop processor cores is no more than 10 or 20 (many are logic cores virtualized by hyper threading technology), and the parallel computing power is far less than that of GPU.
there are many stream processors in GPU of graphics card. For example, the old HD5770 has 800 stream processors, which is equivalent to 800 computing cores. In recent years, the mainstream high-end graphics card has more built-in stream processors and more powerful computing performance. Using GPU for mining operation, the stream processor is used for parallel operation, and the mining program also optimizes the GPU parallel operation, so the operation efficiency is particularly high, and the efficiency of bitcoin mining has been greatly improved
the number of CPU cores is relatively small. At present, the number of the most powerful mainstream desktop processor cores is no more than 10 or 20 (many are logic cores virtualized by hyper threading technology), and the parallel computing power is far less than that of GPU.
3. Graphics card uses GPU's general computing power for mining calculation, and its performance is much better than simple CPU calculation, because GPU has a large amount of concurrency
the professional mining machine does not use GPU, but a special mining chip, which is an ASIC. This kind of chip can only be used for mining, and has no other functions, but it is highly efficient in its own work and faster than GPU.
the professional mining machine does not use GPU, but a special mining chip, which is an ASIC. This kind of chip can only be used for mining, and has no other functions, but it is highly efficient in its own work and faster than GPU.
4. Copied:
in short, mining is to use the chip to carry out a calculation related to the random number, and get the answer in exchange for a virtual coin. Virtual currency can be exchanged for the currency of each country through some way. The stronger the computing power, the faster the chip can find this random answer. Theoretically, the more virtual coins can be proced per unit time. Because it's about random numbers, you can only get rewards if you happen to find the answer. It's possible that one chip will find the answer in the next second, or ten chips won't find the answer in a week. The more chips compute at the same time, the easier it is to find the answer, and the mining machine with built-in multi chips appears. And a number of mining machines to form a "mine" at the same time mining is to improve efficiency. In the mine pool, multiple "self-employed" join an organization to mine together. No matter who finds the answer and digs out the virtual coin, everyone gets the corresponding reward according to the calculation ability of contribution. This way can make the income of "self-employed" more stable
take a popular example:
I write a string of numbers on a piece of paper and give some hints. Whoever guesses correctly will be given a bonus (mining)
smart people can make more guesses (calculation ability)
some people pay money to invite many people to come back and guess together (mining)
some people call everyone together to guess, no matter who guesses, Distribute the bonus (mine pool) according to the proportion of guessing times of each person
the rest is too long, you can search by yourself...
in short, mining is to use the chip to carry out a calculation related to the random number, and get the answer in exchange for a virtual coin. Virtual currency can be exchanged for the currency of each country through some way. The stronger the computing power, the faster the chip can find this random answer. Theoretically, the more virtual coins can be proced per unit time. Because it's about random numbers, you can only get rewards if you happen to find the answer. It's possible that one chip will find the answer in the next second, or ten chips won't find the answer in a week. The more chips compute at the same time, the easier it is to find the answer, and the mining machine with built-in multi chips appears. And a number of mining machines to form a "mine" at the same time mining is to improve efficiency. In the mine pool, multiple "self-employed" join an organization to mine together. No matter who finds the answer and digs out the virtual coin, everyone gets the corresponding reward according to the calculation ability of contribution. This way can make the income of "self-employed" more stable
take a popular example:
I write a string of numbers on a piece of paper and give some hints. Whoever guesses correctly will be given a bonus (mining)
smart people can make more guesses (calculation ability)
some people pay money to invite many people to come back and guess together (mining)
some people call everyone together to guess, no matter who guesses, Distribute the bonus (mine pool) according to the proportion of guessing times of each person
the rest is too long, you can search by yourself...
5. First of all, we analyze the development of China's economy from the perspective of GDP. The GDP is national purchase + government purchase + investment + export import. Since the reform and opening up, China's GDP growth has been mainly driven by investment and export, and the proportion of export stations is larger. In recent years, China's frequent investment and construction of large-scale projects mainly rely on government purchase as another driving force for development. Due to the deformity and bias of domestic instrial structure, national purchase has not been the main economic driving force. China is the country with the highest degree of foreign dependence in the world (the ratio of international trade deficit to national income). According to the textbook, it accounts for 70% (I think it should be only 40% at most), which is nearly twice as high as Russia, the second highest country. Therefore, e to the high degree of foreign dependence, the market main body of made in China is not at home, It has caused the deformity of the national instrial chain Most of the exported nails, steel, can't encourage the public to buy them home, and can't eat them In order to get rid of this dilemma, the government has been calling for the expansion of domestic demand, because the annual trade deficit makes China hoard too much foreign exchange... Ordinary people can't use US dollars to shop in their daily life at home, so the money can only be exchanged by the state treasury in the end.) coupled with the foreign "mping" lawsuits against China and the emergence of trade barriers, the weakness of excessive dependence on foreign countries has begun to hurt China's economy. Economic transformation is not easy. First of all, you have to make rational use of the surplus proction. At this time, the most useful real estate appears, and an investment target that can pull more than 30 instries by itself appears. In addition, land development and auction have become a large part of local government revenue, Therefore, the state vigorously supports (there are many forced demolition problems). But with the passage of time, the overheating of real estate also involves financial instability. When real estate is no longer a necessity but a speculative investment, it will eventually become pure concrete rather than stable repayment ability (whether it is building or buying a house, most businesses and indivials are through loans or mortgages), China's "subprime" crisis began to break out. In fact, according to the theory of western economics, it may be a miracle that China's real estate did not cause China's economy to collapse now, but the state's macro-control and slow weakening of real estate development have indeed played an incomparable role. With the development of economy and the improvement of employment rate in our country, people are happy to see that our GDP is developing at a high speed of 10% every year, and we are jubilant to become the second largest economy in the world. At the same time, we have encountered three headache problems
first, China has become the world's creditor. The foreign exchange reserves of 200 billion US dollars mean that China has more US dollar cash in its treasury than the United States. When its economy goes wrong, it will reach out to China and borrow it. It will return it, but it is just fresh money directly from the banknote printing machine. So what's the difference between the fresh banknotes and the old banknotes? The difference lies in their depreciation and sharp decline in purchasing power. We can think that they originally had 100 cattle, issued 1 million US dollars, and China had 100000 in its pocket. At this time, 10000 US dollars is a cow; Later, the United States and China borrowed the 100000, and when they returned it, they printed another 100000 US dollars to China, but only 11000 US dollars could buy a cow at that time, so they cheated us... Then we can't borrow it. Your loss in China is even more miserable. If I don't lose a dollar against your RMB, you will lose 200 billion RMB. Can you afford to lose? So China had to help him. In this way, China should be very passive. But last year, our leaders began to use US dollars to inhale a lot of gold, turning piles of waste paper into lovely, gorgeous and golden gold. Although we have got rid of the fixed exchange rate system, the stability of the value of gold is far higher than that of paper money. Therefore, comrades, Get some gold before we run out of dollars... At least I made a small profit from my gold
Second, the rapid economic development and high employment rate bring about high inflation rate. With the increase of inflation rate and CPI, more and more people work and have a stable income. Moreover, the income graally increases, and the rise of prices can not be avoided. I still remember that when I was in junior high school, a finished noodle at my home only sold for 3.5 yuan. This time I returned home and asked for 6 yuan. God... Compared with the rise of the exchange rate, it's a good choice for everyone to study abroad. Therefore, how to control the priceless rise and stabilize the inflation is one of the main problems of China's economy in the next step, and this environment also urges us to prepare to invest abroad (we can also be foreign enterprises in foreign countries). Of course, it is not necessarily the European and American powers, but the relatively backward but resource rich small countries, such as Myanmar, Laos, Cambodia, Vietnam and other Southeast Asian countries, whose economic situation and resource utilization rate are very similar to those of China in the 1980s and 1990s, are also encouraging foreign investment. Our great motherland saw such an opportunity, so it vigorously developed the Southwest (this is related to the third issue), established bridgeheads, laid the Pan Asian Railway and so on, and achieved initial results, such as vegetable oil exchange in Yunnan and Thailand, China and Thailand free of fruit and vegetable tariffs and so on... Because I'm here
thirdly, the gap between the rich and the poor is too large, the utilization rate of resources is low, the utilization of resources is unreasonable, etc.
the rapid development makes some people richer and some people may be poorer. To some extent, we have no conscience can be called the law of the jungle, but we can also have a conscience to say that many rich people are not benevolent. This seems to have nothing to do with economy, but it is closely related to social stability and political strength. If these two points are gone, what we can consider is to escape from the country or resell arms to make a fortune. The first to become rich should be those engaged in import and export (both legal and illegal), then those engaged in mining, and those engaged in real estate recently. Here we want to see the times and the conditions for them to become rich. Import and export have just started the reform and opening up. Whether it's the low price of Chinese made goods or the curiosity of Chinese people about foreign goods, these people almost became the first to get rich at that time by using the price difference in the middle. They have a condition to get rich. They have a place to do import and export for you. For bulk logistics, trains can't go directly, Airplanes are too expensive. They can only be transported by sea... The coastal people laughed (so that up to now, some people think that apart from the coastal areas, most of the inland areas are poor, and my mother-in-law should be counted as one...), Later, foreigners found that your mine was better. All right, those who found the mine laughed. Overnight, they changed from farmers to mine owners. Some of them could not adapt to this gap, making mine owners a derogatory term today... For them, wealth and their cultural quality may not be in direct proportion, which leads to some very rampant rich second generation XX. Mine boss mine boss in the final analysis, you have to find your own mine, or you are likely to be a miner... Therefore, it is necessary to have regional characteristics. Finally, when it comes to real estate, it's finally a bit balanced, at least not regional, but you still need to have "resources". Although some of their black hearted people are still poor, from the perspective of economics, I would like to pay tribute to them, because economy is the rational use of all resources to maximize, they not only use the funds, technology, human resources, but also "contacts" resources, Finally, it also affected the economic situation and financial trend of the whole country. A few people own resources, and they don't share all the resources with everyone. Therefore, the gap between the rich and the poor is graally widening, causing a series of problems. In Thailand, the rich people donate money to temples every month, buy things to temples, and then distribute them to the poor and the needy. In addition, temples actively build schools and really achieve compulsory ecation (I was a teacher in Temple schools), I hope the rich in our country can do the same. It's not economics, it's sociology, but I'm still itching to say, remember that the founding of the people's Republic of China was e to grandfather Mao's fight against local tyrants and division of land, be careful of his reincarnation
China's national strength is becoming stronger and stronger, and the strong economy has also brought us a happy life, although the economic system still has many disadvantages (such as buying bonds, not to mention making profits, even losing the principal... But I think with the development of the times, China will only become more and more powerful (if we don't become the second Song Dynasty in history, and find the time to beat those Filipinos who say we Chinese are small eyes every day). God bless China
I have little talent and little learning. I dare to talk about China's economy. If there is something wrong, please correct it and communicate with my friends. If the landlord adopts my question, I would be very grateful
first, China has become the world's creditor. The foreign exchange reserves of 200 billion US dollars mean that China has more US dollar cash in its treasury than the United States. When its economy goes wrong, it will reach out to China and borrow it. It will return it, but it is just fresh money directly from the banknote printing machine. So what's the difference between the fresh banknotes and the old banknotes? The difference lies in their depreciation and sharp decline in purchasing power. We can think that they originally had 100 cattle, issued 1 million US dollars, and China had 100000 in its pocket. At this time, 10000 US dollars is a cow; Later, the United States and China borrowed the 100000, and when they returned it, they printed another 100000 US dollars to China, but only 11000 US dollars could buy a cow at that time, so they cheated us... Then we can't borrow it. Your loss in China is even more miserable. If I don't lose a dollar against your RMB, you will lose 200 billion RMB. Can you afford to lose? So China had to help him. In this way, China should be very passive. But last year, our leaders began to use US dollars to inhale a lot of gold, turning piles of waste paper into lovely, gorgeous and golden gold. Although we have got rid of the fixed exchange rate system, the stability of the value of gold is far higher than that of paper money. Therefore, comrades, Get some gold before we run out of dollars... At least I made a small profit from my gold
Second, the rapid economic development and high employment rate bring about high inflation rate. With the increase of inflation rate and CPI, more and more people work and have a stable income. Moreover, the income graally increases, and the rise of prices can not be avoided. I still remember that when I was in junior high school, a finished noodle at my home only sold for 3.5 yuan. This time I returned home and asked for 6 yuan. God... Compared with the rise of the exchange rate, it's a good choice for everyone to study abroad. Therefore, how to control the priceless rise and stabilize the inflation is one of the main problems of China's economy in the next step, and this environment also urges us to prepare to invest abroad (we can also be foreign enterprises in foreign countries). Of course, it is not necessarily the European and American powers, but the relatively backward but resource rich small countries, such as Myanmar, Laos, Cambodia, Vietnam and other Southeast Asian countries, whose economic situation and resource utilization rate are very similar to those of China in the 1980s and 1990s, are also encouraging foreign investment. Our great motherland saw such an opportunity, so it vigorously developed the Southwest (this is related to the third issue), established bridgeheads, laid the Pan Asian Railway and so on, and achieved initial results, such as vegetable oil exchange in Yunnan and Thailand, China and Thailand free of fruit and vegetable tariffs and so on... Because I'm here
thirdly, the gap between the rich and the poor is too large, the utilization rate of resources is low, the utilization of resources is unreasonable, etc.
the rapid development makes some people richer and some people may be poorer. To some extent, we have no conscience can be called the law of the jungle, but we can also have a conscience to say that many rich people are not benevolent. This seems to have nothing to do with economy, but it is closely related to social stability and political strength. If these two points are gone, what we can consider is to escape from the country or resell arms to make a fortune. The first to become rich should be those engaged in import and export (both legal and illegal), then those engaged in mining, and those engaged in real estate recently. Here we want to see the times and the conditions for them to become rich. Import and export have just started the reform and opening up. Whether it's the low price of Chinese made goods or the curiosity of Chinese people about foreign goods, these people almost became the first to get rich at that time by using the price difference in the middle. They have a condition to get rich. They have a place to do import and export for you. For bulk logistics, trains can't go directly, Airplanes are too expensive. They can only be transported by sea... The coastal people laughed (so that up to now, some people think that apart from the coastal areas, most of the inland areas are poor, and my mother-in-law should be counted as one...), Later, foreigners found that your mine was better. All right, those who found the mine laughed. Overnight, they changed from farmers to mine owners. Some of them could not adapt to this gap, making mine owners a derogatory term today... For them, wealth and their cultural quality may not be in direct proportion, which leads to some very rampant rich second generation XX. Mine boss mine boss in the final analysis, you have to find your own mine, or you are likely to be a miner... Therefore, it is necessary to have regional characteristics. Finally, when it comes to real estate, it's finally a bit balanced, at least not regional, but you still need to have "resources". Although some of their black hearted people are still poor, from the perspective of economics, I would like to pay tribute to them, because economy is the rational use of all resources to maximize, they not only use the funds, technology, human resources, but also "contacts" resources, Finally, it also affected the economic situation and financial trend of the whole country. A few people own resources, and they don't share all the resources with everyone. Therefore, the gap between the rich and the poor is graally widening, causing a series of problems. In Thailand, the rich people donate money to temples every month, buy things to temples, and then distribute them to the poor and the needy. In addition, temples actively build schools and really achieve compulsory ecation (I was a teacher in Temple schools), I hope the rich in our country can do the same. It's not economics, it's sociology, but I'm still itching to say, remember that the founding of the people's Republic of China was e to grandfather Mao's fight against local tyrants and division of land, be careful of his reincarnation
China's national strength is becoming stronger and stronger, and the strong economy has also brought us a happy life, although the economic system still has many disadvantages (such as buying bonds, not to mention making profits, even losing the principal... But I think with the development of the times, China will only become more and more powerful (if we don't become the second Song Dynasty in history, and find the time to beat those Filipinos who say we Chinese are small eyes every day). God bless China
I have little talent and little learning. I dare to talk about China's economy. If there is something wrong, please correct it and communicate with my friends. If the landlord adopts my question, I would be very grateful
6.
Main differences between CPU and GPU:
1. CPU is the central processing unit of computer
GPU is the graphics processor of computer CPU is a very large-scale integrated circuit, which includes Alu arithmetic logic unit, cache memory and bus CPU is the core of a computer's control and operation, its main function is to interpret the instructions issued by the computer and process the big data in the computer software GPU is the abbreviation of image processor. It is a kind of microprocessor which is specially used for PC or embedded device to perform image operation The work of GPU is similar to that of CPU mentioned above, but not entirely. It is designed to perform complex mathematical and geometric calculations. This game has high requirements in this respect, so many game players also have deep feelings for GPU 
so CPU and GPU are two completely different things, they just sound the same
extended data:
CPU and GPU are different in design because they are used to deal with different tasks at first, and some tasks are similar to the problems that GPU is used to solve at first, so we use GPU to calculate. The operation speed of GPU depends on how many pupils are employed, and the operation speed of CPU depends on how powerful professors are employed, The professor's ability to deal with complex tasks is very good, but for less complex tasks, it still can't hold many people
of course, today's GPU can also do some slightly complicated work, which is equivalent to upgrading to the level of junior high school students and senior high school students, but it still needs CPU to feed data to the mouth before it can start to work, which is still managed by CPU
7. Maybe the building owner went to a new server, or the legendary "ghost clothes", where the number of alliance and tribe is extremely uncoordinated. Go to an old server, such as "daslema" in District 2. This server is stable and has the same number of allies and tribes.
8. The application of Web harvest
1. Background
in the era of unprecedented information explosion, people no longer worry about the lack of information, but pay a lot of price for screening useful information. So how to collect useful information? Now there are RSS, blog and other services, but they can not fully meet our needs, because a lot of information is not provided in the form of formatted data, so smart engineers came up with the method of accurate search, resulting in a large number of vertical search sites (such as cool News), which is really popular. Of course, we can't know how they are implemented, but we can also achieve this kind of accurate collection. The open source web harvest is a similar technology, which has been contacted before, so it's written to share with you< Web harvest is an open source web data extraction tool written in Java. It provides a way to extract useful data from the required pages. To achieve this, you may need to use techniques such as XSLT, XQuery, and regular expressions to manipulate text / XML. Web harvest mainly focuses on HMLT / XML based page content, which is still the majority at present. On the other hand, it can easily extend its extraction capability by writing its own Java methods
the main purpose of Web harvest is to strengthen the application of existing data extraction technology. Its goal is not to create a new method, but to provide a better way to use and combine existing methods. It provides a set of processors to process data and control flow. Each processor is regarded as a function, which has parameters and returns results after execution. In addition, in order to facilitate data operation and reuse, web harvest also provides variables above and below to store declared variables
the execution results of the above processes can be stored in the files created ring execution or used in the context of programming< Each extraction process is defined in one or more XML based configuration files and described as specific or structured XML elements. For better explanation, a configuration file is listed below:
& lt; config charset=”gbk”>< br />
<!– Start by crawling the page and search by the keyword: "toys" – & gt< br />
< var-def name=”start” >< br />
< html-to-xml>< br />
< HTTP url = "toys" / & gt< br />
</ html-to-xml>< br />
</ var-def>< br />
<!– Get the list of enterprise websites for bidding ranking – & gt< br />
< var-def name=”urlList” >< br />
< xpath expression=”//div[@class=' r']“& gt;< br />
< var name=”start”/>< br />
</ xpath>< br />
</ var-def>< br />
<!– Loop urllist and write the result to XML file – & gt< br />
< file action=”write” path=”/catalog.xml” charset=”utf-8″>< br />
<![ CDATA[ < catalog> ]]& gt;< br />
< loop item=”item” index=”i”>< br />
< list$amp;& gt; amp;$ lt; var name=”urlList”/$amp;& gt; amp;$ lt;/ list>< br />
< body>< br />
< xquery>< br />
< xq-param name=”item” type=”node()”$amp;& gt; amp;$ lt; var name=”item”/$amp;& gt; amp;$ lt;/ xq-param>< br />
< xq-expression$amp;& gt; amp;$ lt;![ CDATA[
declare variable $item as node() external;< br />
let $name := data($item//span/font[1]/text()[1])
let $url := data($item//span/font[2]/text())
return
< website>< br />
< name>{ normalize-space($name)}</ name>< br />
< url>{ normalize-space($url)}</ url>< br />
</ website>< br />
]]$amp;& gt; amp;$ lt;/ xq-expression>< br />
</ xquery>< br />
</ body>< br />
</ loop>< br />
<![ CDATA[ </ catalog> ]]& gt;< br />
</ file>< br />
</ config>
the above configuration file contains three sections< The implementation steps of the first paragraph are as follows:
1<
3.
the execution steps of the second paragraph:
1. Extract the search results from the given URL with XPath expression
2. Use a new variable "urllist" to save the above search results< In the third paragraph, we use the search results of the previous paragraph to extract the corresponding information:
1
2. Get the name and URL of each item
3. Save it in the file system
with the configuration file (save the configuration file as. XML), let's go one step further and write a few lines of code:
Import java.io.ioexception< br />import org.webharvest.definition.ScraperConfiguration;< br />import org.webharvest.runtime.Scraper;< br />
public class Test {
public static void main(String[] args) throws IOException {
ScraperConfiguration config = new ScraperConfiguration(”c:/.xml”);< br /> Scraper scraper = new Scraper(config, “c:/tmp/”);< br /> scraper.setDebug(true);< br />
long startTime = System.currentTimeMillis();< br /> scraper.execute();< br /> System.out.println(”time elapsed: ” + (System.currentTimeMillis() - startTime));
}
}
let's execute it and see the result:
& lt; catalog>< br />
< website>< br />
< name> Shanghai Liqiang professional & lt/ name>< br />
< url$amp;& gt; amp;$ lt;/ url>< br />
</ website>< br />
< website>< br />
< name> Diversified large & lt/ name>< br />
< url$amp;& gt; amp;$ lt;/ url>< br />
</ website>< br />
< website>< br />
< name> Dr. Tong Cartoon & lt/ name>< br />
< url$amp;& gt; amp;$ lt;/ url>< br />
</ website>< br />
< website>< br />
< name> Sesame Street & lt/ name>< br />
< url> c4</ url>< br />
</ website>< br />
< website>< br />
< name> Dr. Tong, China's affordable student supplies/ name>< br />
< url$amp;& gt; amp;$ lt;/ url>< br />
</ website>< br />
< website>< br />
< name> Inflation & lt/ name>< br />
< url$amp;& gt; amp;$ lt;/ url>< br />
</ website>< br />
< website>< br />
< name> Find wood & lt/ name>< br />
< url$amp;& gt; amp;$ lt;/ url>< br />
</ website>< br />
< website>< br />
< name> Midody & lt/ name>< br />
< url> b14</ url>< br />
</ website>< br />
</ catalog>
isn't it cool. Reptiles are that simple< I don't know if you are familiar with the above configuration, code and results. Is it similar to the way Java reads database data through ibatis
is it possible for us to implement such a mechanism, taking the whole Internet as our huge database, which we can read at will<
Web harvest provides a scrapercontext in which Java objects can be set and corresponding result data can be collected through Java objects (for example, map can be set and data can be collected through map)
scraper provides such methods:
scraper. Getcontext(). Put ("resultdataset", new resultdataset())
resultdataset is a Java object that collects data
then we can do this:
A) first set the path of the web page to visit
scraper. Getcontext(). Put ("startpageref", "; WD = children's toys ")
b) in the second step, set the container to collect the returned data
scraper. Getcontext(). Put ("resultdataset", new resultdataset())
C) data can be set in the configuration file in this way
${resdataset. AddRecord ("SearchResult", "totalsearchresult", totalsearchresult)}
d) after the crawling operation is completed, the data can be returned:
resultdataset resultdataset = (resultdataset) scraper. Getcontext(). Get ("resdataset")
OK, we can use these data at will. Please see the attachment for details< (3) pagination mechanism processing
A) introction
now there is a large amount of information, which is displayed through pagination
A) implementation mechanism
then how do we deal with it? To extract data by paging, we have to clarify several things
1. Pager processing, such as page number, page size, record number or page number<
2. The structure of "next page" address
3. The crawling of data on each page
the paging mechanism of different websites is different, how do we deal with it? Of course, we can't deal with it by hard coding, we just use the configuration file of Web harvest
the configuration file structure of Web harvest itself is:
& lt; config charset=”gbk”>
configuration information
& lt/ config><
extend this structure:
& lt; web-harvest-config>< br />
<!– Generate pager configuration – & gt< br />
< config charset=”gbk” id=”pagination”>
configuration information
& lt/ config>< br />
<!– Assemble next page address – & gt< br />
< config charset=”gbk” id=”urlnav”>
configuration information
& lt/ config>< br />
<!– Grab list data – & gt< br />
< config charset=”gbk” id=”listData”>
configuration information
& lt/ config>< br />
</ web-harvest-config>
we can deal with it through three config items
L in the first step, the pager is generated through the configuration of id = "pagination"
L in the second step, the URL of the next page is constructed through the generated pager and the configuration of id = "urlnav"
L in the third step, First, the advantages and disadvantages of Web harvest
advantages:
lweb harvest is a convenient API library for capturing information, which is currently version 1.0
lweb harvest has good expansibility and can be used as long as the configuration file is modified<
L
disadvantages:
L processing process is more, the corresponding speed is slower
2
1. Background
in the era of unprecedented information explosion, people no longer worry about the lack of information, but pay a lot of price for screening useful information. So how to collect useful information? Now there are RSS, blog and other services, but they can not fully meet our needs, because a lot of information is not provided in the form of formatted data, so smart engineers came up with the method of accurate search, resulting in a large number of vertical search sites (such as cool News), which is really popular. Of course, we can't know how they are implemented, but we can also achieve this kind of accurate collection. The open source web harvest is a similar technology, which has been contacted before, so it's written to share with you< Web harvest is an open source web data extraction tool written in Java. It provides a way to extract useful data from the required pages. To achieve this, you may need to use techniques such as XSLT, XQuery, and regular expressions to manipulate text / XML. Web harvest mainly focuses on HMLT / XML based page content, which is still the majority at present. On the other hand, it can easily extend its extraction capability by writing its own Java methods
the main purpose of Web harvest is to strengthen the application of existing data extraction technology. Its goal is not to create a new method, but to provide a better way to use and combine existing methods. It provides a set of processors to process data and control flow. Each processor is regarded as a function, which has parameters and returns results after execution. In addition, in order to facilitate data operation and reuse, web harvest also provides variables above and below to store declared variables
the execution results of the above processes can be stored in the files created ring execution or used in the context of programming< Each extraction process is defined in one or more XML based configuration files and described as specific or structured XML elements. For better explanation, a configuration file is listed below:
& lt; config charset=”gbk”>< br />
<!– Start by crawling the page and search by the keyword: "toys" – & gt< br />
< var-def name=”start” >< br />
< html-to-xml>< br />
< HTTP url = "toys" / & gt< br />
</ html-to-xml>< br />
</ var-def>< br />
<!– Get the list of enterprise websites for bidding ranking – & gt< br />
< var-def name=”urlList” >< br />
< xpath expression=”//div[@class=' r']“& gt;< br />
< var name=”start”/>< br />
</ xpath>< br />
</ var-def>< br />
<!– Loop urllist and write the result to XML file – & gt< br />
< file action=”write” path=”/catalog.xml” charset=”utf-8″>< br />
<![ CDATA[ < catalog> ]]& gt;< br />
< loop item=”item” index=”i”>< br />
< list$amp;& gt; amp;$ lt; var name=”urlList”/$amp;& gt; amp;$ lt;/ list>< br />
< body>< br />
< xquery>< br />
< xq-param name=”item” type=”node()”$amp;& gt; amp;$ lt; var name=”item”/$amp;& gt; amp;$ lt;/ xq-param>< br />
< xq-expression$amp;& gt; amp;$ lt;![ CDATA[
declare variable $item as node() external;< br />
let $name := data($item//span/font[1]/text()[1])
let $url := data($item//span/font[2]/text())
return
< website>< br />
< name>{ normalize-space($name)}</ name>< br />
< url>{ normalize-space($url)}</ url>< br />
</ website>< br />
]]$amp;& gt; amp;$ lt;/ xq-expression>< br />
</ xquery>< br />
</ body>< br />
</ loop>< br />
<![ CDATA[ </ catalog> ]]& gt;< br />
</ file>< br />
</ config>
the above configuration file contains three sections< The implementation steps of the first paragraph are as follows:
1<
3.
the execution steps of the second paragraph:
1. Extract the search results from the given URL with XPath expression
2. Use a new variable "urllist" to save the above search results< In the third paragraph, we use the search results of the previous paragraph to extract the corresponding information:
1
2. Get the name and URL of each item
3. Save it in the file system
with the configuration file (save the configuration file as. XML), let's go one step further and write a few lines of code:
Import java.io.ioexception< br />import org.webharvest.definition.ScraperConfiguration;< br />import org.webharvest.runtime.Scraper;< br />
public class Test {
public static void main(String[] args) throws IOException {
ScraperConfiguration config = new ScraperConfiguration(”c:/.xml”);< br /> Scraper scraper = new Scraper(config, “c:/tmp/”);< br /> scraper.setDebug(true);< br />
long startTime = System.currentTimeMillis();< br /> scraper.execute();< br /> System.out.println(”time elapsed: ” + (System.currentTimeMillis() - startTime));
}
}
let's execute it and see the result:
& lt; catalog>< br />
< website>< br />
< name> Shanghai Liqiang professional & lt/ name>< br />
< url$amp;& gt; amp;$ lt;/ url>< br />
</ website>< br />
< website>< br />
< name> Diversified large & lt/ name>< br />
< url$amp;& gt; amp;$ lt;/ url>< br />
</ website>< br />
< website>< br />
< name> Dr. Tong Cartoon & lt/ name>< br />
< url$amp;& gt; amp;$ lt;/ url>< br />
</ website>< br />
< website>< br />
< name> Sesame Street & lt/ name>< br />
< url> c4</ url>< br />
</ website>< br />
< website>< br />
< name> Dr. Tong, China's affordable student supplies/ name>< br />
< url$amp;& gt; amp;$ lt;/ url>< br />
</ website>< br />
< website>< br />
< name> Inflation & lt/ name>< br />
< url$amp;& gt; amp;$ lt;/ url>< br />
</ website>< br />
< website>< br />
< name> Find wood & lt/ name>< br />
< url$amp;& gt; amp;$ lt;/ url>< br />
</ website>< br />
< website>< br />
< name> Midody & lt/ name>< br />
< url> b14</ url>< br />
</ website>< br />
</ catalog>
isn't it cool. Reptiles are that simple< I don't know if you are familiar with the above configuration, code and results. Is it similar to the way Java reads database data through ibatis
is it possible for us to implement such a mechanism, taking the whole Internet as our huge database, which we can read at will<
Web harvest provides a scrapercontext in which Java objects can be set and corresponding result data can be collected through Java objects (for example, map can be set and data can be collected through map)
scraper provides such methods:
scraper. Getcontext(). Put ("resultdataset", new resultdataset())
resultdataset is a Java object that collects data
then we can do this:
A) first set the path of the web page to visit
scraper. Getcontext(). Put ("startpageref", "; WD = children's toys ")
b) in the second step, set the container to collect the returned data
scraper. Getcontext(). Put ("resultdataset", new resultdataset())
C) data can be set in the configuration file in this way
${resdataset. AddRecord ("SearchResult", "totalsearchresult", totalsearchresult)}
d) after the crawling operation is completed, the data can be returned:
resultdataset resultdataset = (resultdataset) scraper. Getcontext(). Get ("resdataset")
OK, we can use these data at will. Please see the attachment for details< (3) pagination mechanism processing
A) introction
now there is a large amount of information, which is displayed through pagination
A) implementation mechanism
then how do we deal with it? To extract data by paging, we have to clarify several things
1. Pager processing, such as page number, page size, record number or page number<
2. The structure of "next page" address
3. The crawling of data on each page
the paging mechanism of different websites is different, how do we deal with it? Of course, we can't deal with it by hard coding, we just use the configuration file of Web harvest
the configuration file structure of Web harvest itself is:
& lt; config charset=”gbk”>
configuration information
& lt/ config><
extend this structure:
& lt; web-harvest-config>< br />
<!– Generate pager configuration – & gt< br />
< config charset=”gbk” id=”pagination”>
configuration information
& lt/ config>< br />
<!– Assemble next page address – & gt< br />
< config charset=”gbk” id=”urlnav”>
configuration information
& lt/ config>< br />
<!– Grab list data – & gt< br />
< config charset=”gbk” id=”listData”>
configuration information
& lt/ config>< br />
</ web-harvest-config>
we can deal with it through three config items
L in the first step, the pager is generated through the configuration of id = "pagination"
L in the second step, the URL of the next page is constructed through the generated pager and the configuration of id = "urlnav"
L in the third step, First, the advantages and disadvantages of Web harvest
advantages:
lweb harvest is a convenient API library for capturing information, which is currently version 1.0
lweb harvest has good expansibility and can be used as long as the configuration file is modified<
L
disadvantages:
L processing process is more, the corresponding speed is slower
2
9. Some interesting places in Wenzhou:
Wenzhou is close to the mountains and the sea. It is known as "the best landscape in Southeast China". There are many famous mountains and beautiful scenery in Wenzhou. National key scenic spots include Yandang Mountain and Nanxi River; National nature reserves include Wuyanling and Nanji Island; There are 9 provincial scenic spots, including Xianyan, Yaoxi, Zeya, Binhai yucangshan, Dongtou, zhailiaoxi, zhanggu, 12 municipal scenic spots and 11 county-level scenic spots. The total land area of the scenic spot is 2279 square kilometers, accounting for 20% of the city's land area. There are also 5 national forest parks and 8 provincial forest parks in the city<
Wenzhou has a long history and developed culture. It is known as "Zou Lu in Southeast China" and has rich historical and cultural relics. There are 39 national cultural relics and historic sites, including stone shed tomb, Liuji temple, zhenghetang kiln site, Guanyin temple stone pagoda, Yeshi tomb, and 474 municipal cultural relics< Yandang Mountain (including Beiyan, Zhongyan and Nanyan)
located in the northeast of Yueqing City, beiyandang mountain is famous for its unique peaks, rocks, waterfalls and caves, covering an area of 450 square kilometers. It is named for its "lake at the top of the hill, reeds, grass and wild geese in autumn". The main scenic spots include Lingfeng, Lingyan, dalongqiu, sanzhefu waterfall, Yanhu, xianshengmen, yangjiaodong, etc., with more than 500 scenic spots. Lingfeng, Lingyan and dalongqiu are known as the "three wonders of Yandang"< The Nanxi River is located in Yongjia County, covering an area of 625 square kilometers. It is famous for its beautiful water, strange rocks, many waterfalls and beautiful beaches and forests. The main scenic spots include Daruoyan, shiyan, shiziyan, sihaishan, Yanjiang ancient village cultural area with Yantou as the center, with more than 800 scenic spots< Wuyanling is located in the northwest of Taishun County, with an area of 1902600 hectares and a forest coverage rate of 98%. There are many kinds of trees, 21 of which are listed as precious plants under national protection; 342 species of animals and more than 200 species of birds have been found, belonging to 50 species of national first and second level protected animals and 29 species of birds, among which the yellow bellied Tragopan is the only one in the world< Wuyanling is located 30 nautical miles away from the mouth of Aojiang River in Pingyang, and consists of 23 islands with a total land area of 12 square kilometers. There are 421 species of shellfish, 174 species of algae and 397 species of fish in the sea on the island, which is known as the "blue sea fairy mountain" and "Kingdom of shellfish and algae"< Yueqing is a good place for tourists in Wenzhou and its surrounding areas because of its beautiful landscape, convenient transportation, numerous scenic spots, concentrated scenic spots and unique delicacies<
the Yandang goat horn Cave Scenic Spot, which integrates peaks, peaks, caves, waterfalls, streams, pools and lakes, is a beautiful scenic spot with fantastic mountains and waters. The whole tour generally takes only 2 hours, which is very suitable for weekend self-help tour. Tourists can take the shuttle bus from Xinnan station to Yandangshan, get off at Yandangshan station (Baixi), transfer along national highway 104 to Taizhou, and get off at yangjiaodong scenic spot. In Yandang, besides yangjiaodong scenic spot, there are many good places for self-help tours. For example, Yanhu scenic spot, located on the top of Yanshan mountain, covers an area of about 59 square kilometers. It is the cradle of Yandang Mountain and an ecological reserve<
it is also worth mentioning the Bailong mountain scenic spot in Yueqing. The scenic spot is located in Hongqiao and Danxi towns of Yueqing, covering an area of about 50 square kilometers and 716 meters above sea level. It is famous for its unique stones, including Jiuzhou rock, immortal chair, Tianmen array, baojianyan, cuttlefish rock, the top of Longshan mountain, tenglongling, etc. Self help tour is also very convenient. Starting from Wenzhou City, tourists can take a bus at Xinnan station to Yueqing Hongqiao, and then transfer from Hongqiao to Bailong mountain
you have to try Puqi's shark feast when you go to Yueqing. Shark feast is Puqi's main feature: shark skin, shark bone, shark whip, shark meat, shark belly, shark lip and other parts of the shark can be processed into dishes. Puqi ding'anmen restaurant is one of the more representative. There is a Puqi exit on Yong Tai Wen expressway, which is very convenient for both self driving and shuttle bus.
Wenzhou is close to the mountains and the sea. It is known as "the best landscape in Southeast China". There are many famous mountains and beautiful scenery in Wenzhou. National key scenic spots include Yandang Mountain and Nanxi River; National nature reserves include Wuyanling and Nanji Island; There are 9 provincial scenic spots, including Xianyan, Yaoxi, Zeya, Binhai yucangshan, Dongtou, zhailiaoxi, zhanggu, 12 municipal scenic spots and 11 county-level scenic spots. The total land area of the scenic spot is 2279 square kilometers, accounting for 20% of the city's land area. There are also 5 national forest parks and 8 provincial forest parks in the city<
Wenzhou has a long history and developed culture. It is known as "Zou Lu in Southeast China" and has rich historical and cultural relics. There are 39 national cultural relics and historic sites, including stone shed tomb, Liuji temple, zhenghetang kiln site, Guanyin temple stone pagoda, Yeshi tomb, and 474 municipal cultural relics< Yandang Mountain (including Beiyan, Zhongyan and Nanyan)
located in the northeast of Yueqing City, beiyandang mountain is famous for its unique peaks, rocks, waterfalls and caves, covering an area of 450 square kilometers. It is named for its "lake at the top of the hill, reeds, grass and wild geese in autumn". The main scenic spots include Lingfeng, Lingyan, dalongqiu, sanzhefu waterfall, Yanhu, xianshengmen, yangjiaodong, etc., with more than 500 scenic spots. Lingfeng, Lingyan and dalongqiu are known as the "three wonders of Yandang"< The Nanxi River is located in Yongjia County, covering an area of 625 square kilometers. It is famous for its beautiful water, strange rocks, many waterfalls and beautiful beaches and forests. The main scenic spots include Daruoyan, shiyan, shiziyan, sihaishan, Yanjiang ancient village cultural area with Yantou as the center, with more than 800 scenic spots< Wuyanling is located in the northwest of Taishun County, with an area of 1902600 hectares and a forest coverage rate of 98%. There are many kinds of trees, 21 of which are listed as precious plants under national protection; 342 species of animals and more than 200 species of birds have been found, belonging to 50 species of national first and second level protected animals and 29 species of birds, among which the yellow bellied Tragopan is the only one in the world< Wuyanling is located 30 nautical miles away from the mouth of Aojiang River in Pingyang, and consists of 23 islands with a total land area of 12 square kilometers. There are 421 species of shellfish, 174 species of algae and 397 species of fish in the sea on the island, which is known as the "blue sea fairy mountain" and "Kingdom of shellfish and algae"< Yueqing is a good place for tourists in Wenzhou and its surrounding areas because of its beautiful landscape, convenient transportation, numerous scenic spots, concentrated scenic spots and unique delicacies<
the Yandang goat horn Cave Scenic Spot, which integrates peaks, peaks, caves, waterfalls, streams, pools and lakes, is a beautiful scenic spot with fantastic mountains and waters. The whole tour generally takes only 2 hours, which is very suitable for weekend self-help tour. Tourists can take the shuttle bus from Xinnan station to Yandangshan, get off at Yandangshan station (Baixi), transfer along national highway 104 to Taizhou, and get off at yangjiaodong scenic spot. In Yandang, besides yangjiaodong scenic spot, there are many good places for self-help tours. For example, Yanhu scenic spot, located on the top of Yanshan mountain, covers an area of about 59 square kilometers. It is the cradle of Yandang Mountain and an ecological reserve<
it is also worth mentioning the Bailong mountain scenic spot in Yueqing. The scenic spot is located in Hongqiao and Danxi towns of Yueqing, covering an area of about 50 square kilometers and 716 meters above sea level. It is famous for its unique stones, including Jiuzhou rock, immortal chair, Tianmen array, baojianyan, cuttlefish rock, the top of Longshan mountain, tenglongling, etc. Self help tour is also very convenient. Starting from Wenzhou City, tourists can take a bus at Xinnan station to Yueqing Hongqiao, and then transfer from Hongqiao to Bailong mountain
you have to try Puqi's shark feast when you go to Yueqing. Shark feast is Puqi's main feature: shark skin, shark bone, shark whip, shark meat, shark belly, shark lip and other parts of the shark can be processed into dishes. Puqi ding'anmen restaurant is one of the more representative. There is a Puqi exit on Yong Tai Wen expressway, which is very convenient for both self driving and shuttle bus.
10.
The analysis is as follows:
1. The center of Wenzhou is in Fuqian street and Jiefang street
Wenzhou is a prefecture level city under the jurisdiction of Zhejiang Province; It is one of the regional central cities in Zhejiang Province. Wenzhou is located in the southeast of Zhejiang Province, on the South Bank of the lower reaches of Oujiang River. The land area of the city is 11612.94 square kilometers, and the sea area is about 11000 square kilometers. The city has jurisdiction over four municipal districts, five counties and two county-level cities, with a permanent resident population of 9.215 million in 2017 Wenzhou is a middle subtropical monsoon climate zone, with significant alternation of winter and summer monsoon, moderate temperature, four distinct seasons and abundant rainfall. The annual average temperature is 17.3-19.4 ℃, the average temperature in January is 4.9-9.9 ℃, and the average temperature in July is 26.7-29.6 ℃ There was no severe cold in winter and no severe heat in summer. The annual precipitation is 1113-2494mm. There are Meiyu at the turn of spring and summer, tropical cyclones from July to September, and frost free period is 241-326 days. The annual sunshine hours are 1442-2264 hoursHot content